Building products to scale environmental Life Cycle Assessment (LCA) leveraging data & AI

Took a career break to catch up on a long list of todo items around personal hobbies, travel and learning. Then ventured deeper into climate change solutions and worked on AgTech startup ideas to drive changes to agricultural practices to help mitigate the problem.

I took on the role of a manager to help with the rapid growth of Hortonworks. In Jan 2019, the two biggest rivals in the Hadoop/Open source big data space merged and I continued my journey as part of Cloudera.Leadership contributions: * Helped the startup scale, growing the team from 2 to 20+ and leading multiple new projects * Led critical parts of the data warehouse portfolio, which was responsible for more than half of $800m+ annual revenue * Led efforts to create a robust open source community around the products. Leveraged the efforts to build a strong recruitment pipeline * Led several projects successfully that had significant impact on major customers adoption and retention.Product contribution highlights: * Cross data center replication for data warehouse for disaster recovery. Redesigned core aspects of Hive ACID as part of this.* Providing access to Spark jobs to Hive ACID tables and Ranger fine grained access control* Hive metastore development - Scale/Performance/Consistent caching* JDBC/ODBC connectivity support - Scale/Performance/Security/SSO for public Cloud* Query federation - optimized access of data in other RDBMS* DataWarehouse PaaS offering in AWS and Azure - Upgrade functionality for runtime and K8s* Development of new execution engines Apache Tez & Hive LLAP, ORC storage layer and Iceberg layout (Architecture/Design inputs)

I was part of the founding team of engineers from Yahoo that created Hortonworks when it started in July 2011.* Created the first versions of the first product (Hortonworks Data Platform) * Worked on multiple aspects of Apache Hive including Client Connectivity, Query Compiler and Query Execution* Developed a new version of JDBC/ODBC connectivity service and made it ready for enterprise adoption* Developed REST API service for Hive for Metadata access and Job submission* Created new secure implementation of Hive Authorization

http://pig.apache.org/

Working on design and development a highly scalable parallel SQL query engine that runs on commodity hardware, for the PetaByte proprietery datawarehouses at Yahoo.Architected changes and led parts of the team in several project cycles which added features and made significant improvement in the performance.Designed and implemented components in all parts of the query engine including optimizer and core execution engine.Interacted closely with the users to understand their OLAP requirements and use cases, with the purpose of helping management align project priorities and making architecture changes to maximize impact on business needs.

Enabling job search at yahoo. http://search.yahoo.com/jobs* Designed and implemented software applications for enabling job search at Yahoo! Hotjobs.* Wrote algorithms and heuristics to identify job pages in websites and extract job information out of the job pages.* Implement the pipeline that takes webpages crawled by the web-crawler, passes it through the core extractor to extract job features, stores the information in a staging database, applies business rules on the extracted content in the staging database, checks for dead linksand loads the content into the job listings database.* Designed the database schema, tuned the performance of the MySQL database and troubleshot several database issues..

* Designed and implemented a query engine with an SQL like interface for yahoo proprietary user database.* Designed and developed an application for identifying user interests based on their activity in yahoo network.* Developed several data feeds that summarized data from yahoo traffic logs.* Helped in preparing data warehouse team's blue sky vision document.* Researched ways for storing data in compressed files with indexes in them. * Created a prototype for creating compressed files with indexes and retrieving data from them. Studied the performance improvements.* Wrote a data warehouse access and storage tool for yahoo proprietary data warehouse.