Data Engineer
• Experience in migrating existing legacy applications into Optimized data pipelines utilizing Spark w and Python with testability and perceptibility.• Developed Optimized and tuned ETL operations in Azure Data Factory.• Developed ETL solutions using SSIS, Azure Data Factory and Azure Data Bricks.• Worked on Talend integrations to ingest data from various sources into Data Lake.• Worked on migration On Prem Databases on to Azure Cloud with appropriate business transformations.• Extensive hands-on experience on Azure GIT Repos• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like SQL Server, Blob storage, Azure SQL Data warehouse.• Extensively worked on Copy, Foreach, get meta data, look up activities, Data flows, Azure Functions and notebooks.• Implemented the Change Data capture and email notifications using Azure Logic Apps.• Performed unit testing for all the ADF pipelines and published the pipelines to master branch by raising pull request.• Automated the deployments using Azure Pipelines and automated the test plan execution using Azure Test Plans.• Developed the Dashboards and analytical reports using Power BI.• Extensively worked on Self hosted integration run time issues, Data Quality Issues and Key Vault issues• Handled all kind of data formats including Parquet, Avro, Json and Delimited files.• Practiced Azure Boards for Project management and deliverables.• Experience in Developing Spark applications using Spark - SQL in Azure Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.• Responsible for estimating the cluster size, monitoring, and troubleshooting of the Hadoop cluster.