Azure Data Engineer
Current1. Developed and managed end-to-end data pipelines using Azure Data Factory for ingesting, transforming and loading (ETL) data, improving data availability by 40%.2. Automated data pipeline scheduling, monitoring, and alerts using ADF to ensure reliability and timely data delivery.3. Leveraged Databricks and Apache Spark jobs for data transformations, optimizing the data processing pipeline for 50 GB daily data ingestion.4. Applied partitioning, bucketing, and caching techniques in Spark jobs to optimize performance and reduce shuffle time.5. Implemented data validation checks, ensuring high data quality and integrity across the entire data pipeline.6. Utilized Azure monitoring tools to continuously track performance, identifying and resolving bottlenecks in Azure Data Factory and Spark jobs, reducing incidents by 25%.