Senior Data Engineer
Current- Established a PostgreSQL data warehouse with automated ETL processes via Fivetran
- Implemented DBT (Data Build Tool) pipelines for all data warehouse tasks, ensuring efficient and reliable data transformations
- Deployed Metabase open source instance and connected the data warehouse to facilitate self-service analytics and automate BI reporting
- Advanced infrastructure as code (IaC) adoption using Terraform, facilitating scalable and consistent infrastructure provisioning and management
- Developed and maintained various robust, fault-tolerant web scraping solutions with mechanisms to handle site changes, bot detection, dynamic content loading, and rate limits