Senior Data Engineer
Current• Worked closely with Business partners, cutomer and BSA's to understand business requirements, perform analysis, and design quality technical solutions that comply with the organization's architectural standards.• Responsible for Designing, Coding, Testing and Maintaining data pipelines in Azure Data Factory that are within scope based on established high-quality Data Engineering standards.• Designed and implemented ETL pipelines to process semi-structured data by integrating data from different sources to load in Enterprise Datawarehouse.• Developed complex queries involving analytic functions and stored procedures to perform data aggregations and data readiness• Accountable for troubleshooting production issues, and data analysis and participated in technical code reviews, design and comprehensive documentation for enhancements and Production defects.• Automated Data pre-validation and automated data cleanup for SparkSql jobs using Python scripting.• Built data masking custom Python utility to mask confidential customer data for Performance testing in PAT environment.• Implemented Custom logging and premeptive alerts across data pipelines to make sure data ingestion is reliable and correct.• Developed data quality checks by implementing automated data validation Data Bricks notebooks, resulting in a 90% reduction in data errors before sending data to Business.• Developed and maintained code using DevOps tools like GIT for version control, Bitbucket, XLR tool for CI/CD deployment to Azure cloud platform.• Developed ETL spark jobs in ADF scheduled through orchestration framework and created complex SQL queries for production defects.
