Staff Software Engineer
CurrentI've been responsible for helping the team bringing Datadog's next-generation storage system to production. Crucial contributions to this effort include: I conceived of and implemented the "stateless" deduplication scheme at ingest--which made it possible to take on our first large internal customers. I conceived of and implemented a means to enforce data locality (similar to Snowflake micropartitions) where a best-effort schema and data ordering is established over a table at ingest and extended across all data shards at compaction. This reduced overall system cost by at least 30% by allowing early pruning of data during queries. Releasing this storage system and turning down the older one represents a 5x improvement in cost for Datadog's most data-intensive products, such as logs. I continue to explore improvements such as delayed materialization, plus additional large-scale analytical features supporting our most important products.