Staff Data Engineer
Current- Architected and implemented Airflow DAG management framework enabling complex workflow logic and actionable alerting.
- Designed data pipeline that automatically handles file structure changes; eliminating the need for manual updates and configurations.
- Lead developer in developing ETL pipeline using Python, Airflow, Glue and AWS Lambdas; enabling quick onboarding of 1,000+ customers.
- Subject-matter-expert with several critical services, training teammates as appropriate to reduce knowledge silos.
- Optimized SQL queries, indices and data structures for performance and scalability.
- Maintained infrastructure as code (IaC) with Terraform, Docker and Jenkins.