Sr Software Engineer
CurrentLoad raw data from DSO from multiple tables, transform them as per requirement in data bricks and load them into destination using ADF.Extracted data from multiple sources like CSV files, .txt files, SQL server and transformed the data using different transformations like dataconversion, Derived columns, look up, Conditional Split, Aggregate, Union all, merge join,cast transformations etc.Have used Spark SQL, PySpark in Azure Data Bricks to transform Big Data.Extensively used transformation functions like Filter, withColumn, Distinct, Drop Duplicates, FillNA, Union , Group By , Rank , Dense Rank etc. to transform data in data bricks notebook .• Sound knowledge on performance optimization of the spark jobs.Sending Success automated Email notification provided with all details like pipeline name, activityname, file rows read count, file rows copied count.Sending failure automated Email notification provided with all details like pipeline name, activityname, failure reason, as well file input location.Implemented Azure Logic Apps to automate those emails to send notifications to the users.Implemented logic to dynamically delete the old Run ID folders.Have used Azure Data Bricks for the statistical analytics of huge data.Designed packages Using Control Flow Tasks like For Loop Container, For Each Loop Container,Sequential Container, Execute SQL Task and Data Flow Task.
