Data Engineer Ii
Stockholm, Stockholm County, Se
Spearheaded the implementation of sort merge bucket (SMB) optimizations for one of the most widely used datasets at Spotify, mitigating Dataflow shuffle costs, decreasing overall runtime by 15%, and enabling more efficient joins for hundreds of downstream consumers.Developed Scio pipeline to process 100+ TB's of licensor data through utilization of monoids and aggregators. Empowered gross profit predictions.Drove refactoring of core Python and Scala abstractions through inheritance and mix-in functionality to DRY up code, resulting in a 25% line reduction and increasing test coverage to 96%.Instructed DE 201: Introduction to Data Engineering.Taught programming languages, software architecture, and pipeline development to aspiring engineers in the 2022 NYC Tech Fellowship program.Designed and implemented offline click through rate (CTR) feature construction through Scio batch pipeline, enabling a ranking model to be trained at the uri-country and uri-query granularity.