Senior Data Engineer
Current Strategic ownership over the data architecture, data models and ETL processes used to build out the foundation for our analytical tool Built ETL processes in Azure Data Factory (ADF) to integrate clients new and historical data set into data mart Review and Optimize Query Performance, Coding and Monitoring Overall DWH Performance Designed and implemented a real-time data pipeline (AZ Data bricks) to process semi-structured data by integrating larger volume of raw records from various data sources into Data Lakes using PySpark Designed the data pipeline architecture for a new product that quickly scale up the daily transactions Utilized Spark in Python to distribute data processing on large streaming datasets to improve ingestion and processing speed of that data by 67% Imported large volume of data using Poly Base in a Dedicated SQL Pool using Azure Synapse Analytics Studied and revamped data dictionaries to include a robust history for developing consistency across domain Performed and documented data Analysis, data validation, data mapping, create prototypes and data samples, Generate Adhoc Reports Design, Develop and maintaining documentation of Data Architecture, Data flows and Data Models Responsible for overall quality assurance and deliverables (Functional and Non Functional Testing) Provided guidance on best practices and approaches to design and implement effective and Standardized driven dashboards and analytics