Azure Date Engineer
CurrentProject Description: Developed and maintained data pipelines and storage systems. Transformed SQL Stored Procedures into Azure Data Flows and Pipeline activities. Created and tested Data Pipelines for ETL procedures. Utilized ADF tasks workflows, including task assignments, version controls on Azure DevOps, and accomplishments. Collaborated with stakeholders to provide technical solutions.Roles and responsibilities: · Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse · Optimized and deployed Python web applications using Azure DevOps CI/CD for streamlined development. · Designed and implemented streaming data pipelines using Apache Spark Structured Streaming, Kafka, or Azure Event Hubs, integrating them with Delta Lake to ensure data consistency and reliability in near real-time scenarios. · Created data integration and technical solutions for Azure Data Lake Analytics, Azure Data Lake Storage, Azure Data Factory, Azure SQL databases, and Azure SQL Data Warehouse for providing analytics.
