Staff Data Engineer
Current• Led a corporate intitive to transition from SQL Server and SSIS into cloud technologies.• Designed and built end-to-end data pipelines using S3, Snowpipe, Confluent Kafka, Snowflake, dbt and Airflow to replace hosted Apache Kafka, SQL Server stored procedures and SSIS packages.• Coached team members on cloud technologies, such as AWS, Snowflake, dbt Cloud and MWAA.• Designed and built data vault models using Snowflake and dbt.• Improved ETL run time from 9 hours to 2 minutes using dbt incremental models, Snowflake query optimization and clustering for better micro-partition pruning.• Created python DAGs in Airflow MWAA and Astronomer to orchestrate dbt jobs, as well as for data loading and monitoring.• Developed custom timetable plugins and operators in Astronomer to expand Airflow capabilities.