Data Engineer Ii
CurrentDeveloped batch big data pipelines in Scala and Python using Scio, Luigi, and Flyte running on GCP Dataflow.Created datasets that were used for surfacing insights to creators on Spotify for Artists.Collaborated with machine learning engineers on the implementation of pipelines to produce feature data for model training and batch inference.Combined system telemetry with product data to derive metrics for monitoring the performance of forecasting models.Collected and centralized event data output from multiple microservices to gain a deeper understanding of system states and their impact on the health of the product.