Data Lakehouse ManagementUsing pySpark, Spark SQL, Azure Data Factory, and SQL Server Change Data Capture (CDC), engineered and maintained a Data Lakehouse in Azure Databricks.Spearheaded a proof of concept (POC) leveraging Delta Live Tables and SQL Server Change Data Capture (CDC) to establish near real-time data ingestion from an application database, significantly enhancing data freshness.OrchestrationThrough a combination of Databricks Workflows and Azure Data Factory Pipelines, created an orchestration process that ensures data is available in BI tool (Power BI) as soon as it’s processed.Databricks Asset BundlesSource controlled Databricks jobs and deployed via Azure DevOps, ensuring they were built, run, and tested in QA prior to being deployed.

SQL Server Development:• Create and modify Stored Procedures backing CRM and other applications, and work with Software Developers to ascertain data needs and develop database solutions to meet them (stored procedures, functions etc).SSIS:• Create data flows and write C# script tasks to complete ETL processes, make API calls, and automate reports. Accomplishments:• Successfully architected database schema and developed stored procedures to support the Route Optimization application built in house.

ETL and Report Automation(Python SSIS SQL):• Numerous Python scripts in production to ingest, manipulate, and transform data obtained from vendor reports, SQL Databases and API’s.Through automated SSIS packages, currently distributing 40+ daily reports and dashboards using data extracted from vendor reports, Data mart, API’s, MS Access etc, and finally queried in SQL Server. Application Development:• .NET Framework(C#, VB.NET) -- Windows Form Apps (WF) used by Analysts to update SSIS connections for move to production server, and SQL Server connections embedded in Excel files, and for personal use updating Control table responsible for maintaining all automated reports.• Responsible for the creation and automation of Report distribution:Through automated SSIS packages and Python scripts, currently distributing 40+ daily reports and dashboards. Data gathered via vendor reports, DataMart, API’s, MS Access etc. Reports distributed via Excel and Tableau.Accomplishments:• Created a Python script to translate a Dialer Campaign schedule into a format that could be pushed to a SQL Database allowing us to monitor our campaigns as compared to plan.• Automated entire Natural Disaster Reporting regimen via a combination of SSIS packages, and Python scripts.

• Assisted in the automation of compliance reviews using SQL queries to pull information previously retrieved manually from the system of record (MSP), and VBA macros to pull information stored in Excel Files• Wrote scripts (vbscript) to efficiently retrieve populations of loans to review, saving time and virtually eliminating the potential for user error• Worked closely with SME's to ensure controls were effective and up to date