Experienced Data Engineer with a proven track record in designing and implementing scalable data pipelines. Proficient in ETL processes, data modelling, and database management, with strong programming skills in Python, SQL and PySpark. Adept at optimizing data infrastructure to enhance performance and reliability. Familiar with cloud platforms, particularly Azure, and skilled in leveraging cloud services for data engineering tasks. Detail-oriented and committed to delivering high-quality, actionable insights through effective problem-solving and innovative solutions.Projects Undertaken (Key Responsibilities):• Value Share Forecasting:-Perform EDA to understand trends, seasonality, and patterns in the data within the Databricks.Segment the data by relevant categories such as product lines, geographical regions, customer demographics, etc., to refine the analysis.Analyzed and understood the data architecture of complex systems,• Promotion Analytics:-Design, implement, and manage ETL pipelines using Azure Data Factory (ADF) and Databricks to process and integrate promo data from various sources.Implement data validation and quality checks within ETL processes to ensure the accuracy, completeness, and consistency of promo data.Collaborate with data scientists and analysts to develop and refine promo models, ensuring accurate and reliable data inputs.Developed the entire pipeline using Azure Data Factory to ensure seamless integration and automation.Work closely with cross-functional teams and clients to understand data requirements and provide analytical support for promotional strategies.• Data Integration and Data Warehousing:-Design, implement, and manage data pipelines using Azure Synapse Analytics to facilitate data integration from various sources into a central data warehouse.Plan and execute data integration strategies, ensuring the accurate transfer of data from legacy systems to data warehouse using Azure Synapse Analytics.Implement data validation and quality checks within data integration processes to ensure the accuracy, completeness, and consistency of integrated data.implemented automated data pipeline scheduling to ensure timely data processing across multiple environments, using tools like Azure Data Factory.
