Data Engineer
- Led ELT transformation, migrating from on-premises SQL Server to Azure Data Lake Storage(Gen2),leveraging Databricks’ Unity Catalog feature.
- Established bronze and silver layers in Databricks(Medallion Architecture) for efficient data processing andtransformation, implementing SCD type 2 logic.
- Introduced YAML parameterization for optimized code flexibility and maintenance, enhancing workflowefficiency.
- Championed Databricks Cluster optimization, reducing job cluster startup times through adept instance poolutilization.
- Utilized REST APIs for comprehensive metadata tracking and detailed job run logging, ensuring accountability.
- Orchestrated dynamic pipeline architecture in Azure Data Factory for automated workflow checks andadaptive execution.