Azure Data Engineer
CurrentResponsibilities:• Analyze, design, and construct modern data solutions utilizing Azure PaaS services to facilitate data visualization. Comprehend the current production status of the application and assess the impact of new implementations on existing business processes.• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services- (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.• Collaborate with data architect/engineers to establish data governance/cataloging for MDM/ security (key vault, network security schema level & row level), resource groups, integration runtime setting, integration patterns, and aggregated functions for data bricks development.• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backward.• Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into customer usage patterns.• Expertise in Creating, Debugging, Scheduling, and Monitoring jobs using Airflow and Oozie.• Created data bricks notebooks using Python (PySpark), Scala, and Spark SQL for transforming the data that is stored in Azure Data Lake stored Gen2 from Raw to Stage and Curated zones.• Built numerous technology demonstrators using Confidential Edison Arduino shield using Azure EventHub and Stream Analytics, integrated with Power BI and Azure ML to demonstrate the capabilities of Azure Stream Analytics.• Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that processes the data using the SQL Activity.poses.