• Worked as data engineer developer• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.• Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark databricks cluster.• Doing nested JSON files transformation with mapping dataflow and databricks notebooks and loading data into sql tables• Hands-on experience on developing SQL Scripts for tables creation and removing duplicates

Azure Data Engineer with Datafactory, Databricks and Pyspark.• Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks• Developing scalable and re-usable frameworks for ingesting of datasets• Integrating the end to end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times• Working with event based / streaming technologies to ingest and process data• Databricks level code deployments doing from one branch to another branch• SQL DB migration from on-premise to the PaaS MI and Azure SQL DB with DMA and DMS tool• Delta load has performed in migration with ADF• Created IR, LS, Data Set and Pipelines according to requirement and transform the data from Azure blob storage/Azure Data Lake storage to Azure SQL Database in Delimitated form data.• Worked to move the data transform from On-Premises SQL database table to Azure SQL database using ADF• Create automated workflows with the help of triggers• Worked on different Variables like Set, Append and conditional and looping statements in ADF.• Building the pipelines to copy the data from source to destination in Azure Data Factory• Migrating the data from different sources to the destination with the help of ADF