Data Engineer
CurrentRevamped ETL/ELT data pipelines to new infrastructure using data task orchestration via Airflow and data modeling within DBT, resulting in more maintainable pipelinesExperienced in interacting with APIs, SFTPs, and various cloud storage platformsRegularly utilize Git and GitHub for version control and team collaborationMigrated Airflow DAGs using Exavault SFTP to Filemage, saving $600 per monthEnhanced Airflow dedupe operator by adding time window parameters, resulting in major cost savingsRound-the-clock monitoring/troubleshooting of data pipelines to ensure customer satisfactionDeveloped internal API endpoints using Python and FastAPI to retrieve data from PostgresSQL, facilitating its integration within a product