Worked as a part of Big Data Org:1. Co-founded a low latency project, fault-tolerant, scalable, reliable system aims to provide the Hadoop NameNode, yarn ResourceManager, Hive Metastore, etc. logs in a consumable format near real time (latency of few seconds). Several products internally leverage these low latency logs to serve the customers faster.2. Worked on the GDPR Compliance and data privacy. - Redesign the pipeline to process anonymized data. - Migrating the pipeline on cloud service which involves redesigning various components. - Make the pipeline more scalable to handle data growth without violating SLA. - Enable pipeline to support new dataset formats.

Worked on the TuneIN framework: a framework that automatically tune Hadoop jobs to save cluster resources. Did the POC on optimization of execution time of hadoop jobs via this framework. Achieved overall 25-35% improvement in execution time.