Data Engineer
▪ Led ELT transformation, migrating from on-premises SQL Server to Azure Data Lake Storage(Gen2),leveraging Databricks’ Unity Catalog feature.▪ Established bronze and silver layers in Databricks(Medallion Architecture) for efficient data processing andtransformation, implementing SCD type 2 logic.▪ Introduced YAML parameterization for optimized code flexibility and maintenance, enhancing workflowefficiency.▪ Championed Databricks Cluster optimization, reducing job cluster startup times through adept instance poolutilization.▪ Utilized REST APIs for comprehensive metadata tracking and detailed job run logging, ensuring accountability.▪ Orchestrated dynamic pipeline architecture in Azure Data Factory for automated workflow checks andadaptive execution.▪ Initiated strategic migration of SSIS packages to Databricks, facilitating data dashboarding and analysis.▪ Automated routine tasks using Python scripts, improving team productivity and reducing errors.▪ Introduced Checkpointing functionality for gold tables, ensuring swift data recovery in case ofpipeline failures.▪ Orchestrated various tasks for efficient ELT loads using Databricks Workflows functionality.▪ Established audit table for comprehensive metadata tracking of pipelines and job executions, enhancingaccountability.
