Senior Data Architect/Engineer
Architect-ed, designed and built state-of-the-art cloud-based Data Lakehouse built on Azure Databricks, Azure Data Factory, Azure Data Lake Storage and Azure KeyVault. Data Lakehouse built on medallion architecture leveraging custom metadata-driven ELT pipelines in Azure Data Factory to query/extract/store all in-coming "raw" data in Azure Data Lake Storage and created custom Azure Databricks pipelines for loading in data into "bronze/silver" Delta Live tables with options for row-level history tracking. Integrated roughly 60 different internal and third-party data sources containing roughly 2000 separate near-real-time datasets into the Data Lakehouse. Worked on data migration from legacy Data Warehouse into the new Data Lakehouse.Implemented data collection and dashboards to track cloud resource utilization and spend. Based on those trends, implemented design optimizations through smart-triggered pipeline integrations across Azure Data Factory and Azure Databricks piplines to drastically reduce costs by more than 50%. Worked directly with Databricks engineers to address any performance, security and scaling issues.Worked with data supplier and data consumer teams across the company to deprecate any Data Warehouse process or reporting dependencies.Worked directly with auditors and security teams to establish SOX access, change-management and operational controls to maintain compliance within the Data Lakehouse.