Staff Data Engineer
Current- Led a corporate intitive to transition from SQL Server and SSIS into cloud technologies.
- Designed and built end-to-end data pipelines using S3, Snowpipe, Confluent Kafka, Snowflake, dbt and Airflow to replace hosted Apache Kafka, SQL Server stored procedures and SSIS packages.
- Coached team members on cloud technologies, such as AWS, Snowflake, dbt Cloud and MWAA.
- Designed and built data vault models using Snowflake and dbt.
- Improved ETL run time from 9 hours to 2 minutes using dbt incremental models, Snowflake query optimization and clustering for better micro-partition pruning.
- Created python DAGs in Airflow MWAA and Astronomer to orchestrate dbt jobs, as well as for data loading and monitoring.