Data Engineer
Current- Member of Workspace Analytics responsible for building a logs processing platform that produces canonical usage,performance and experiment metrics for Google Workspace which processes 3 trillion events and tens of petabytes every day.- Prototyped an architecture to transition from batch to streaming pipelines which allowed the master data to be available a few hours earlier.- Led work to design and implement release best practices for data pipelines (resources, permissions, monitoring, storage, testing, scheduling and compliance) which lowered production outages by 30% and doubled weekly releases.- Migrated Apache Beam data pipelines (Go, Java) to new architectures while ensuring data stability (>97% identical) with regression tests and SQL queries.- Optimized storage and runtime of batch data processing pipelines with performance traces.- Distilled ~125 mostly unactionable weekly pagers to ~30 actionable ones by designing an alerting feature taking into account the alerting set up upstream of each pipeline.