Building the next-generation streaming platform

Continuing to lead development on our proprietary dataset. Lead overhaul of data processing rearchitected across the offline and real-time modeling stacks.

Tech lead and primary developer on an in-house analytics engine responsible for delivering realtime audience insights to customers about their web properties and advertising campaigns. Contributions include* Several compounding 5-10x performance improvements.* Greatly increased the size and variety of datasets, now including over a trillion facts.* Replaced over two dozen batch pipelines in favor of interactive computation.* introduced time-series capabilities.

Oversaw a zero-downtime migration of 500+ map-reduce pipelines from a physical datacenter into AWS.Leveraged algorithmic abilities and distributed systems expertise to release a variety of optimizations and improvements to batch computing pipelines and the real-time bidding stack saving hundreds of thousands of dollars annually.

Helped design and implement a novel A/B test framework for giving estimates on effect of experiments to external attribution rate.Helped develop in-house SQL-over-Map Reduce system, including conceiving of and implementing several features to address customer needs.Rearchitected data serialization format for the two most expensive pipelines, achieving a 60% reduction in compute and storage cost