● Migrated traditional ETL processes from on-premises to the AWS Cloud using AWS and PySpark, focusing on loading data into the Teradata warehouse.● Investigated and resolved issues with failed jobs on the AWS EMR Cluster, successfully fixing all problems.● Worked on optimization techniques to enhance the performance of PySpark jobs.● Implemented unit testing for all ETL mappings and addressed defects identified by both the testing and user teams.● Contributed to the migration from Teradata to Snowflake by adapting the existing Spark framework to integrate with Snowflake’s objects and modifying the Spark code to ensure successful data loading into Snowflake's target tables.● Collaborated closely with the testing team to provide an in-depth understanding of the ETL framework codebase, enabling them to perform thorough testing effectively.

● Collaborated with business analysts and data stakeholders to gather data requirements and implemented ETL processes accordingly.● Developed and maintained ETL pipelines using Databricks with the PySpark DataFrame API.● Engaged in data transformation and preprocessing tasks utilizing Snowflake and SQL.● Implemented data validation checks and error handling mechanisms to enhance data quality and accuracy.● Ensured seamless execution of the data pipeline, promptly investigating and addressing any failed tasks.● Occasionally supported my team with code deployment activities in higher environments.● Modified and transformed the existing codebase, transitioning the inventory module from pandas to PySpark, and collaborated closely with data scientists on data transformation and preprocessing scripts while enhancing data models to handle large volumes of data.● Participated in architectural discussions for data migration from on-premises to cloud services, contributing insights into the end-to-end data pipeline design.