Architected scalable ETL solutions using AWS Glue and EMR for processing large datasets and integrating with Redshift.Implemented data migration projects from on-premise databases to AWS, ensuring data integrity and optimizing performance.Creating AWS Lambda functions using python for deployment management in AWS and designed, investigated and implemented public facing websites on Amazon Web Services and integrated it with other applications infrastructure.Designed and implemented ETL workflows using Py Spark and AWS Glue to efficiently process and transform large datasets, reducing data processing times.Creating different AWS Lambda functions and API Gateways, to submit data via API Gateway that is accessible via Lambda function.Utilized Python Libraries like Boto3, NumPy for AWS.Developed serverless data processing workflows using AWS Lambda and S3, reducing operational overhead and improving efficiency.Developed custom Py Spark jobs for batch processing, enabling the processing of millions of records daily and supporting large-scale data analytics.Created external tables with partitions using Hive, AWS Athena and Redshift.Worked closely with stakeholders to define data requirements, design architecture, and deliver solutions that met business objectives.Utilized Py Spark to process and analyze large datasets in a distributed computing environment, significantly reducing the time required for data transformation tasks.Created Py Spark applications for real-time data processing and analytics, integrating with AWS services such as S3 and Redshift to store and query processed data.Automated monitoring and logging using CloudWatch and integrated alerting mechanisms to maintain high availability of data services.Involvement in using collections in Python for manipulating and looping through different user defined objects.

Led the end-to-end data migration from legacy systems to AWS Redshift, optimizing data extraction, transformation, and loading processes.Designed and developed ETL pipelines using Python and AWS services, supporting analytics and reporting requirements.Enhanced AWS environments for performance, security, and cost-efficiency, reducing costs by 20%.Provided technical leadership in cloud data engineering, advising on best practices and strategies.Built and optimized SQL scripts and Python code for data processing and analytics operations on AWS.

Managed end-to-end data migration from legacy systems to AWS Redshift, including data extraction, transformation, and loading.Designed and implemented ETL pipelines using Python and AWS services to support data analytics and reporting needs.Worked with cross-functional teams to optimize AWS environments for performance, security, and cost-efficiency.Developed and maintained SQL scripts and Python code to support data processing and analytics operations on AWS.Provided technical leadership in data architecture, cloud strategy, and best practices for cloud data engineering.Assisted the development team in sending the correct data via query strings using PostgreSQL as the back-end for storing data.Used Python and Django to interface with the jQuery UI and manage the storage and deletion of content.Build SQL queries for performing various CRUD operations like create, update, read and delete.Placed data into JSON files using Python to test Django websites. Used Python scripts to update the content in database and manipulate files.Generated Python Django forms to maintain the record of online users.Generated Django forms to record data of online users and used PyTest for writing test cases.Extensive use of version controlling systems like GIT and SVN.Designed and developed data management system using MySQL.Built application logic using Python 2.7, Django and MYSQL.Rewrite existing Python/Django modules to deliver certain format of data.Adept at automating repetitive work through Shell scripts and Python.Able to analyze a requirement and act accordingly by managing the resources efficiently.

Designed and implemented data pipelines using AWS Glue and AWS Lambda for processing and transforming data from on-premise databases to AWS.Migrated legacy data systems to AWS Redshift, improving query performance and enabling advanced data analytics.Developed Python scripts for data extraction, transformation, and loading (ETL) to automate data workflows and reduce manual intervention.Collaborated with cross-functional teams to gather requirements, design solutions, and deploy data infrastructure on AWS.Developed Spark scripts by writing custom RDDs in Scala for data transformations and perform actions on RDDs.Optimized SQL queries and data models in Redshift to improve data retrieval times and reduce costs.Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.