Leading Cloud & Platform Infrastructure to help set Data in motion

Leading Cloud & Platform Infrastructure to help set Data in motion

To help place an event streaming platform at the heart of every company

My time is divided between people management, operations, technical discussion, strategy planning and hiring. Ambry - Distributed object storeHighly available, scalable and fault tolerant distributed blob storage system. I wrote the code for the initial version of the system and then grew a team of amazing engineers to scale the system to be able to store multiple petabytes of data. In just over a year, the system was built, operationalized, data was migrated and all the clients were moved to the new APIs. We continue to invest in building a world class object store to serve our media use cases. Vector - Media upload, serving, asset management, metadata search and processing. The focus is to build a scalable, efficient and fault tolerant world class media platform that supports images, documents and videos. Media processing has to be near-real time, needs to support elastic capabilities, has to work on large objects and requires reprocessing.

Ambry - Distributed object storeStarted a new team to build a highly available, scalable and fault tolerant distributed blob storage system. I wrote the code for the initial version of the system and then grew a team of amazing engineers to scale the system to be able to store multiple petabytes of data. In just over a year, the system was built, operationalized, data was migrated and all the clients were moved to the new APIs. We continue to invest in building a world class object store to serve our media use cases. Apache Kafka (Distributed message queue) and Samza (Distributed stream processing)I still continue to be actively involved in these projects. I have contributed to different parts of the project including helping in designing the non blocking producer and transactions in Kafka and various features in Samza (pluggable partition grouping, exactly once semantics, config as a stream and standalone processor without Mesos or Yarn). I have also given numerous presentations on Kafka and Samza to different companies during this time.

I helped setup the data pipeline and data integration ecosystem at Linkedin. My time was mainly spent on building Apache Kafka and its surrounding tools. I also worked quite a bit on Apache Samza.Apache KafkaWorked on implementing and stabilizing replication and doing a massive company wide migration to the new system which was backwards incompatible. Also designed and operationalized Kafka REST proxy to provide multi language support across the company. Made major revamp of schema registry to support key features and maintained auditing framework. Worked with clients to set up their data pipeline which included reviewing AVRO schemas, on boarding, metrics/monitoring and auditing. Designed, implemented and stabilized various operational tools include controlled shutdown, adding partitions, auto rebalance, replica reassignment etc. I gave numerous presentations in meet ups and to different companies to spread the knowledge of Kafka, helped the open source use the system and became an Apache committer. Apache SamzaI was part of Samza since January 2013 before it was open sourced. I was involved in the design and implementation of multiple features including metadata caching, throttling, YARN CGroups, message chooser etc. Was deeply involved in proposing multiple strategies to support exactly once semantics with very low performance overhead. We open sourced Samza in Sept 2013.

I spent most of my time building the Hotmail storage platform. I also contributed to the business tier of Hotmail including anti spam, notifications etc. Helped implement, stabilize and deploy data de-duplication across petabytes of data to reduce storage cost. Designed and implemented strategy to preallocate search index files to reduce IO utilization and deploy them in a backwards compatible way across thousands of nodes. Designed and implemented a distributed cache to batch writes to disk for high write throughput. Proposed and implemented a online defragmentation scheme for sharded SQL databases to reduce read IOPS by 10x. Conducted research to study the IO patterns of the system and identified and fixed key bottlenecks. Initiated and implemented the efforts to use Cosmos (distributed data processing) for log aggregation and analysis. Conducted hackathons to build team bonding and create a more agile culture. Mentored multiple interns and team members for their career growth. Software Development Engineer 2 (1) - October 2010 - Aug 2011Software Development Engineer 2 (2) - Aug 2011 - Dec 2012

Worked extensively on designing and implementing Windows Live Movie Maker (Similar to iMovie on the Mac). This was a V1 project and we had to pretty much build everything from ground up. This included working with PMs to design the features, implement and test these features, conduct bug bash to stabilize, participate in user interaction study and help in creating tutorials and on boarding experience. I built the complete stack for text rendering on images and videos including building a WYSIWIG editor to accept text input, apply shaders to create different text effects and position text in the scene. I helped in designing and implementing various controller components to orchestrate movie playback including video transcoding, image resizing, effects and animations. I also designed and implemented video trimming functionality including working as the designer for the feature, I completely led the design and implementation of an auto moviemaking feature based on mining metadata from images and videos. Filed two patents on image and video visualization and rendering.Software Development Engineer (1) - May 2008 - Aug 2009SoftwareDevelopment Engineer (2) - Aug 2009 - September 2010