Senior Data Engineer
CurrentData Lakehouse ManagementUsing pySpark, Spark SQL, Azure Data Factory, and SQL Server Change Data Capture (CDC), engineered and maintained a Data Lakehouse in Azure Databricks.Spearheaded a proof of concept (POC) leveraging Delta Live Tables and SQL Server Change Data Capture (CDC) to establish near real-time data ingestion from an application database, significantly enhancing data freshness.OrchestrationThrough a combination of Databricks Workflows and Azure Data Factory Pipelines, created an orchestration process that ensures data is available in BI tool (Power BI) as soon as it’s processed.Databricks Asset BundlesSource controlled Databricks jobs and deployed via Azure DevOps, ensuring they were built, run, and tested in QA prior to being deployed.