Data Engineer 2
Current- Developed and optimized end-to-end data pipelines using Microsoft Azure, including the extraction of data from SFTP sources, transformation into Parquet format and integrating into Data Lake Storage using Databricks
- Optimized SQL queries to streamline the data transformation processes and implemented the incremental load and full load strategies for data refreshes in the data warehouse enhancing data processing
- Implemented data quality checks and monitoring within Delta Live Tables (DLT) pipelines to ensure real-time data ingestion and processing, improving data availability and analytics capabilities