Building data platform in the cloud.

• Built end to end data platform of Peta byte scale on Apache Big Data stack.• Developed multi-year road map with business stakeholders, technology leads and senior management.• Built data ingestion pipeline for ingesting and querying data in real time. The platform ingested several million messages per day with throughput of 40,000 messages per minute. The messages can be more than 1 MB in size.• Successfully delivered Records Management product.• Designed and developed data storage on HDFS to support bi-temporal views, concurrent reads / writes, schema evolution and circular references in data model.• Built data catalog and API layer to enable data discovery and data exploration.• Implemented controls framework for monitoring, alerting, reconciliation and data quality.• Built a 30 member core team of data engineers based in India and US.

• Built a platform on Spark which generated data with 3 billion rows having 600 columns monthly, approx. 1 TB of compressed data.• Replaced legacy platform on Vertica with 50% run time improvement on the same hardware.• Built predictive ETA application for batch jobs using Spark ML.• Built a streaming platform for users to provide calculation rules and data adjustments through UI application.• Migrated 300+ informatica ETL jobs to Spark.

The SaaS Analytics platform is the data lake for accumulating machine generated data from various sources in an enterprise and applying advanced machine learning techniques to 1. Enable detection of anomalies in real time.2. Prediction of issues in future3. Provide a platform which enables end-users to write intelligent Apps on top of data in the lake.