• Worked closely with Business partners, cutomer and BSA's to understand business requirements, perform analysis, and design quality technical solutions that comply with the organization's architectural standards.• Responsible for Designing, Coding, Testing and Maintaining data pipelines in Azure Data Factory that are within scope based on established high-quality Data Engineering standards.• Designed and implemented ETL pipelines to process semi-structured data by integrating data from different sources to load in Enterprise Datawarehouse.• Developed complex queries involving analytic functions and stored procedures to perform data aggregations and data readiness• Accountable for troubleshooting production issues, and data analysis and participated in technical code reviews, design and comprehensive documentation for enhancements and Production defects.• Automated Data pre-validation and automated data cleanup for SparkSql jobs using Python scripting.• Built data masking custom Python utility to mask confidential customer data for Performance testing in PAT environment.• Implemented Custom logging and premeptive alerts across data pipelines to make sure data ingestion is reliable and correct.• Developed data quality checks by implementing automated data validation Data Bricks notebooks, resulting in a 90% reduction in data errors before sending data to Business.• Developed and maintained code using DevOps tools like GIT for version control, Bitbucket, XLR tool for CI/CD deployment to Azure cloud platform.• Developed ETL spark jobs in ADF scheduled through orchestration framework and created complex SQL queries for production defects.

• Implemented process automation through Oozie workflows, streamlining operations and enhancing productivity.• Established Hive tables and facilitated seamless data migration from HDFS directories, optimizing data management processes.• Assumed responsibility for project support and maintenance, ensuring continuous functionality and reliability.• Successfully led the migration of extraction projects from Hive to Spark, improving data processing capabilities.• Utilized Autosys for efficient job scheduling and GIT, ensuring seamless workflow management.• Engineered a scalable data storage solution leveraging Hadoop Distributed File System (HDFS), resulting in a 10x increase in data handling capacity without compromising performance.• Established Hive tables and facilitated seamless data migration from HDFS directories, optimizing data management processes.• Collaborated with cross-functional teams to design and implement data quality frameworks, improving data accuracy and integrity.• Worked on migrating historical data from on-prem application to cloud.