Project: Fidelity, HealthEM• Engineered and validated high-performance PySpark code in Azure Synapse to construct a scalable and robust ELT data pipeline.• Optimized the processing of over 700 million records, leveraging advanced Delta Lake features and performance tuning strategies to achieve rapid data transformations in Databricks.• Orchestrated complex data workflows in ADF Pipelines, enabling seamless execution of both sequential and parallel tasks for streamlined data processing.• Automated data integrity checks and report validation processes, boosting accuracy, consistency, and operational efficiency across analytics workflows.

Project: JPMChase• Utilized Databricks along with PySpark, Delta Lake, and ADLS Gen2 to enhance data pipeline performance, resulting in a 50% increase in processing efficiency through advanced optimizations and data engineering best practices.• Performed comprehensive unit, integration, and performance testing to ensure robust deployment.• Engaged in daily Agile stand-ups with the Scrum Master and cross-functional teams, driving sprint progress and delivering iterative improvements.• Partnered closely with data scientists and analysts to facilitate seamless data integration, enabling enhanced reporting and advanced analytics capabilities.

Project: Citi Bank• Developed robust PySpark and Python scripts for large-scale data processing on the Hadoop ecosystem, optimizing performance and resource utilization.• Designed and implemented end-to-end ETL pipelines using SQL, Python, PySpark, HDFS, enabling efficient data transformation and analysis of complex datasets.• Administered version control and automated CI/CD workflows with Bitbucket, Jenkins, and JIRA, ensuring seamless code integration and deployment.• Partnered with the Production Support Team to implement and deploy critical fixes in the production environment, maintaining system stability and operational continuity.