Senior Data Engineer
CurrentData engineers maintain services, APIs, and real-time pipelines that serve data to everyone in the organization, from back end services to Snowflake to ML models. Data arrives from our app users and from third party vendors, and is transformed both in batches and in real time using services using Kubernetes, SNS, SQS, Kafka, and Airflow.Major projects:- Refactor primary real-time data pipeline to accept input data from a second, new vendor- Implement new vertically integrated CI/CD method, enabling easier and more thorough testing, significantly improved development and release speeds, and one-click deployment of long-standing or ephemeral copies of applications/infrastructure. Also includes implementation of Bazel to replace Makefiles.- Deploy product to brand new, clean slate AWS account using vertically integrated deployment method. Configure one-click deployments and rollbacks for this product in dev, UAT, production, etc.