Founder member of a new team in Amazon Elastic Map Reduce. Working on unique and interesting problems related to the intersection of cloud and big data.My recent major contributions to EMR:• EMR Notebooks: A managed environment based on serverless Jupyter Notebooks that allows data scientists, analysts, and developers to prepare and visualize data, collaborate with peers, build applications, and perform interactive analysis using EMR clusters.• Notebook-scoped libraries: Ability to distributed-install Python libraries on the EMR cluster at runtime and use it in EMR notebook. No need to recreate the cluster or reattach the notebook to cluster. Installed libraries are scoped to current notebook session hence multiple users with different notebooks can use different set of Python libraries independently without interfering with other. Refer my AWS technical blog linked below for more details.• Application History Service: Separation of application metrics and logs from cluster lifecycle and making them available in near-real-time as a separate service. Customer can access application level metrics and logs in near-real-time for running as well as terminated clusters via EMR console.• Enhanced Debugging: Auto-detection of EMR step failure root cause and exposing it as part of 'describe-step' API. Designed and implemented rule based engine to classify the root cause based on exception stack trace.My open source contributions to Spark:• https://github.com/apache/spark/commit/e20d9b15655ffaf1c220fbad1a2aa1c2d45137ba• https://github.com/apache/spark/commit/0ff67a1cf91ce4a36657c789c0fe676f4f89282f

Big Data Platform: Product Tracking and Analysis teamBuilt an entire product from scratch for near realtime metrics of PayPal transactions using Spark streaming, Kafka and HBase.Designed and implemented modules for cummulative sessionization and dedupe of Paypal transactions.Designed and implemented an API on top of MySQL to store and manage product metadata including kafka offsets.Reduced the latency of getting transaction metrics from 1 day(in existing old solution) to 3 mins.Scaled the product to handle ~1.5 millions events per minute by tuning number of executors, number of partitions and executor memory.Enhanced PayPal behavioral analytics platform which is the main behavior analysis system based on events for all of PayPal. Platform primarily built on Apache Hadoop ecosystem including HDFS, MapReduce, HBase, Pig and Hive. Designed and developed a tool for continuous purging of platform generated old data with configurable data retention.Designed and implemented MapReduce jobs to migrate 2 years worth of platform reference data from version 2.0 to version 3.0.

Assisted Prof. Ellis Horowitz and Prof. Marco Papa.

Assisted Prof. Ellis Horowitz and Prof. Marco Papa.

Elastic Compute Cloud : Reserved Instances teamDesigned and developed Limit Increase feature for EC2 Reserved Instances web service. Designed and developed Pricing Engine web application to generate market prices of AWS EC2 instance types across all regions. Worked extensively on Ruby on Rails, JQuery and JavaScript.

Under guidance of Prof. Bill Cheng.

Messaging and Web Group: Symantec Message Filter teamDesigned and developed Message Audit Log Feature for Symantec Message Filter 6.3 (Antispam- Antivirus Engine). Developed client side component of New Dispositions feature for Symantec Message Filter 7.0. Wrote a configuration migration script in XSL for upgrade Symantec Message Filter 7.0. Got good exposure to network configuration, SSH, TCP/IP, DNS and DHCP. Got exposure to network programming in C, C++ on Windows Server 2k3/2k8, Linux (RHEL 4/5), Solaris x86 and Solaris SPARC (8/9/10). Got exposure to Message Transfer Agents Postfix, IIS.

Designed and implemented a prototype for a secure and efficient peer to peer file sharing mechanism for multi region corporate network. Customized open source bit torrent private tracker, indexer and client. Implemented automatic content delivery mechanism for efficiency.

Designed Online Analytical Processing based multidimensional database schema along with Conceptual, Logical and Physical data models of ‘TCS Gems Employee Recogonition’ application.