Data Operations Engineer
Current- Designed and Optimized ETL Pipelines: Created, managed, and improved ETL pipelines to ingest CCD data XML files from client SFTPs and FTPs into Snowflake databases, utilizing Azure Data Lake storage for efficient data.
- Developed Client Deliverables Using Airflow and Spark: Utilized Airflow DAGs and Spark to generate deliverables based on CCD data for insurance members. Identified and resolved issues within output file pipelines.
- Designed Data Processing Pipelines: Developed pipelines using Python, Snowflake, and SQL Server to extract customer membership data and generate output files for healthcare data vendors. Implemented dynamic automation.
- Upgraded Data Processing Pipeline: Enhanced the pipeline for a new client implementation by incorporating PGP file decryption capabilities using the GnuPG Python library.
- Managed HIPAA-Compliant ETL Data Loading: Executed and monitored ETL processes for loading data into on-premises SQL Servers while ensuring compliance with HIPAA regulations.
- Trained and Mentored New Team Members: Onboarded and guided 6 new hires and 2 interns, providing training on analyzing and enhancing operational data ETL pipelines. Facilitated their development through hands-on.