Data Operations Engineer
Current• Designed and Optimized ETL Pipelines: Created, managed, and improved ETL pipelines to ingest CCD data XML files from client SFTPs and FTPs into Snowflake databases, utilizing Azure Data Lake storage for efficient data processing. • Developed Client Deliverables Using Airflow and Spark: Utilized Airflow DAGs and Spark to generate deliverables based on CCD data for insurance members. Identified and resolved issues within output file pipelines, including debugging Airflow DAGs and Spark jobs on Kubernetes clusters in Azure. • Designed Data Processing Pipelines: Developed pipelines using Python, Snowflake, and SQL Server to extract customer membership data and generate output files for healthcare data vendors. Implemented dynamic automation for contracted clients and ad hoc solutions for one-off data requests. • Upgraded Data Processing Pipeline: Enhanced the pipeline for a new client implementation by incorporating PGP file decryption capabilities using the GnuPG Python library. • Managed HIPAA-Compliant ETL Data Loading: Executed and monitored ETL processes for loading data into on-premises SQL Servers while ensuring compliance with HIPAA regulations. • Trained and Mentored New Team Members: Onboarded and guided 6 new hires and 2 interns, providing training on analyzing and enhancing operational data ETL pipelines. Facilitated their development through hands-on mentorship, ensuring they were equipped to manage and optimize pipeline performance effectively.