6+ years experienced Data Engineer with passion for building end-to-end direct capabilities, writing scalable code, as well as testing and debugging software pipelines required for optimal ETL and ELT of data from wide variety of sources.•Extract data from Blob Storage, Azure Synapse, SQL Server sources using Azure data factory and load it into Data Lake.• Migrated SSIS packages into Azure data factory pipelines.• Used Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data.• Hands on experience in using existing activities Copy, Foreach, get meta data, look up activities, Data flows (sort, aggregate, union, conditional split, join, Lookup, Derived column, rank, filter, pivot, unpivot, etc.)• Hands on experience in using file formats such as csv, excel, Parquet, Json and Delimited files.• Experience in Developing Spark applications using Spark - SQL in Azure Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.• Worked on Triggering the Databricks notebook from ADF.• Used Schedule, Tumbling Window, Storage events triggers.• Worked on the exploratory data analysis.• Designed, built and managed data pipelines for data structures encompassing data transformation, data models, schemas, and metadata and workload management using MS Azure Data Factory, Azure Data Lake Storage, and Azure Data Bricks.• Worked on Data cleansing, Data wrangling, Data aggregation and data transformation.• Hands-on experience on Azure DevOps (Pipelines, Repos, Boards)• Created ADO multistage Pipelines to deploy the code to Test / Pre-Prod / Production environments.• Create pull requests in DevOps and maintain the code in Azure DevOps repositories.
