Data Engineer
CurrentOrchestrating end-to-end ETL processes using Medallion architecture and Azure Data Engineering tools, I have designed and implemented automated data pipelines in Azure Synapse Analytics and Azure Data Factory to efficiently integrate data from diverse source systems into data warehouses, ensuring scalability, performance optimization, and reliable data workflows.Loading data from diverse sources such as Oracle, MS SQL Server, and files into Azure Synapse Analytics serverless SQL Pool, storing it in ADLS Gen2 as Parquet files in the Bronze layer to ensure efficient storage, processing, and scalability.Transforming Parquet files to Delta Lake format in Silver layer for time travel capabilities, utilizing control tables and ingestion history tables to manage versioning and track data changes efficiently.Designed and implemented the Gold layer to meet business needs. Led development of dataflows and PySpark notebooks for data cleaning and transformation, ensuring high-quality, actionable datasets. Optimized data pipelines for performance, reducing query times and improving scalability for large Datasets.Created and optimized complex SQL queries and views on top of the Gold layer to address diverse business use cases, enabling efficient data retrieval and providing actionable insights to drive key business decisions.Implemented data warehouse concepts, including star schema in Dims and Fact tables, and managed Slowly Changing Dimensions (SCD1, SCD2) to track historical data changes and ensure accurate reporting.Collaborated closely with Data Scientists and Data Architects to understand and address business requirements, ensuring alignment between data engineering solutions and analytical needs.Maintain comprehensive migration documentation for inter-environment migrations (e.g., QA to Prod), along with detailed documentation for data architecture and workflows, to ensure smooth collaboration and effective knowledge transfer with the team.