Sr.Automation Engineer
Current Implemented PySpark using Python and utilizing data frames and temporary table SQL for faster processing of data. Integrated Snowpark scripts with various data sources, such as Delta Lake, Parquet, and CSV, to streamline data ingestion and processing workflows. Spearheaded an initiative to optimize database performance by analyzing and refactoring PLSQL code. Developed web applications within a Python and Java environment that includes distributed web servers and a MySQL… Show more Implemented PySpark using Python and utilizing data frames and temporary table SQL for faster processing of data. Integrated Snowpark scripts with various data sources, such as Delta Lake, Parquet, and CSV, to streamline data ingestion and processing workflows. Spearheaded an initiative to optimize database performance by analyzing and refactoring PLSQL code. Developed web applications within a Python and Java environment that includes distributed web servers and a MySQL database cluster. Utilize PyUnit, the Python unit test framework, for all Python applications. Designed UI interfaces using Django Templates, HTML, APP SYNC, CSS and Bootstrap. Developed REST-based Applications using Python, Django, Angular, CSS, HTML, APP SYNC, and Node.js. Designed, developed, and deployed microservices architecture for scalable and modular applications. Developed a fully automated continuous integration system using GIT, Amplify, EMR Gerrit, Jenkins, AWS Glue MySQL and custom tools developed in Python and Bash. Employed optimization techniques in Snowpark to enhance query performance and reduce processing time for data-intensive tasks. Developed data transition programs from Dynamo DB to AWS LAMBDA, snowflake Redshift (ETL Process) using AWS LAMBDA, snowflake, by creating functions in Python for certain events based on use cases. Developed and implemented Snowpark applications on Data bricks for efficient and scalable data processing. Integrated load balancing mechanisms to distribute incoming traffic among microservices for optimal performance. Designed and implemented an automated reporting system using Python and PLSQL, providing timely and accurate reports to stakeholders. Designed a Python script to load transformed data into cloud services AWS LAMBDA, snowflake, Azure. Developed a fully automated continuous integration system using Gerrit, Jenkins, AWS Glue MySQL and GIT, Amplify custom tools developed in Python Show less