Azure Data Engineer
CurrentWorked on developing and enhancing the ingestion framework used for ingesting thedata from dynamic sources to Synapse Dedicated SQL pool.ADF, Databricks, Pyspark, SQL, Synapse etc are the skills used for developing theabove frameworkDeveloped multiple ADF pipelines to ingest data from external sources into ADLS & blobstorage accounts.Data from the above containers is processed through Landing, Loading, History, Certifiedand Reporting zones.Data Validation is implemented while data is moved from Landing to Loading zone.From Loading zone data is stored in the form on delta and is processed across differentlayers as per the business use cases.Implemented various features such as archival, data rehydration, change data capture,creating dynamic views using metadata etc.Worked on creating the RDDs, DFs for the required input data and performed the datatransformations usingꢀ Pyspark and Spark-SQL.Developed multiple stored procedures and functions for inserting and updating data totables.Created multiple tables based on the entity-relationship diagrams of the table columns.Performed Testing the framework through a wheel file version by adding to data brickscluster.Document all the developed stored procedures, adf pipelines along with test values toSend communication to all the customers.Contributes towards Analysis, Design, Development, Production, and implementation ofETL projects in Azure.