• Led cross-functional Agile team of 10+ developers and testers to build and maintain internal automotive demand forecast platform Unity and by extension, its external product FAST. • Managed end-to-end product development lifecycle from inception to deployment. Drove data pipeline and UI development, including architecture, planning, execution, and release management. Managed strategic planning and resolution of security compliance issues and tech debt. • Collaborated with architects, product owners, and development teams to define product roadmap, release plans, break down features, and manage dependencies. Utilized Jira for task management and dependency tracking.• Contributed and oversaw data pipeline development to ensure timely delivery and data accuracy.• Mitigated risks and resolved technical challenges through effective problem-solving and stakeholder management.• Added structure and processes in a start-up like environment to enhance team productivity and product quality. • Facilitated Agile ceremonies (stand-ups, reviews, retrospectives, planning) to ensure team alignment and efficiency. • Coordinated change management and release deployments. • Ensured high-quality delivery through code reviews and quality assurance processes. • Spearheaded hiring efforts for multiple engineering roles screening over 180 resumes and conducted 50+ interviews. Recruited and on-boarded 5 high-performing engineering talent to support project growth. • Eventually coached team as we planned release efforts from 2 weeks to 1-month sprints. • Successfully delivered multiple product releases on time and within budget.• Coordinated environment usage and ensured availability of environments for features/releases/hotfixes to be tested. • Developed and maintained data pipelines for vehicle demand forecasting publication in Python Pandas using AWS Batch jobs, Lambdas, S3 files, RDS for PostgreSQL, Step Functions, SNS & Cloud Watch.

Operationalization of Text Analytics Application for a US Based Insurance Firm - + Operationalized an existing Streamlit based Text Analytics Application that helps train customer care agents to improve customer retention. Converted client modes to cluster mode along with the addition of restartability. + Migrated the codes from On-Prem to Azure and used CI/CD pipelines to deploy the codes in different environments using Azure DevOps and supported successful run of the solution on prod environment.+ Optimized backend queries to improve runtimes. Tested codes with mock datasets and proved that non-partitioned source tables and its Avro file format caused read overheads that prevented further improvement to runtimes.+ Performed data validations from various runs. Analyzed how records categorization as “loser” can be improved by refreshing older data to include transactions over a 90-day window and implemented the same.Inventory Management Dashboard for Fortune 500 American Manufacturer of Industrial Tools and Household Hardware - + Architected and built AWS EMR based data pipelines in Pyspark that feed Hive tables powering highly responsive Tableau Dashboards in order to profile, derive insights and optimize Inventory levels. Designed based on current UI requirement and future enhancements needed for the dashboard.+ Lead 3 other Data Engineers for few months to build and deliver pipelines via 14 scripts that orchestrate 4 views of the dashboard.+ Solved a critical and challenging algorithm in Pyspark for finding out how many weeks inventory will last based on the upcoming cumulative demand. Further expanded the code to handle more corner cases and ensured that no duplicates are produced. Calculated expected results for a relatively large sample using Excel and comparing its results with the ones produced by Pyspark codes to ensure 100% accuracy.+ Performed Spark Optimization to bring the run time to 26-35 minutes from the initial 3 hours 15 minutes.

Inventory Management Dashboard for Fortune 500 American Manufacturer of Industrial Tools and Household Hardware - + Acted as a liaison to a consultant. Facilitate discussions for requirements gathering efforts and performed EDA to identify SAP tables in the AWS Data lake as sources of interest for a Tableau Dashboard. + Created Pyspark scripts for the first version of the dashboard using queries with stage-by-stage validations in order to profile and document issues with data sources. Also performed spark optimizations while building these scripts and designed the final output Hive table consumed by Tableau for faster querying and daily archiving of data. + Created an automated pyspark script that will help validate various tables across regions of databases within new AWS datalake by just changing the list of input tables. Identified, validated, and reported issues with AWS datalake Hive tables with their SAP counterparts. Dashboard to reduce Premium Freight Costs for Fortune 500 American manufacturer of industrial tools and household hardware -+ Optimized and rebuilt Pyspark EMR that feed Hive tables powering Tableau Dashboards in order to help reduce Premium Freight costs. + Integrated with s3 based log and configuration files to productionize the scripts.+ Spark optimizations were implemented for faster ingestion of Historical codes with s3 based logs for easier, faster and more efficient ingestion.+ Added enhancements to the existing solution and fixed bugs.Internal Revenue-based application for an Indian Media Conglomerate - + Redesigned and rebuilt a pre-ingestion pipeline in Scala for different revenue data files. + Developed extensive unit tests for modules to ensure files are uploaded only on success of quality checks and created appropriate logs in case of failure.

Internal Aggregated Television Viewership based application for an Indian Media Conglomerate - + Worked in a Scala-EMR-S3-Redshift based application that consumes aggregated television viewership ratings and helps make business decisions based on viewership habits. + This was an Agile and Test-Driven Development (TDD) based application where we rebuilt several multi-stage processes to ingest parquet data efficiently into S3 for feeding Redshift External Tables. + Built Airflow DAGs to ingest and validate this parquet data, which gets triggered and notified on Slack with pre-validations and post-validations performed on Athena tables and data enrichment performed using EMR. + Designed and added new features like channel attributes, merging of multiple live telecast programmes’ sessions while the channel name and programme information remained in flux over time and validated the same. Designed and streamlined the validation of data on the addition of new features like identifying original programming.

AWS Datalake for Television Viewership data for an Indian Media Conglomerate - + Built an AWS Data lake by building efficient automated big data pipelines to ingest and validate aggregated viewership data from different S3 sources using Spark SQL, Python, Postgres DB, and UNIX Shell Scripts. Scaled-down data size from TB scale XLSX or CSV data to GB scale orc data for faster and efficient querying in Hive. + Recreated a precise algorithm on aggregating individual viewership data using Spark to match the expected aggregated viewership by cracking the nuances of a black box-like algorithm and saving up to $10k per annum

Spark-SQL-Performance TPCDS testing on medium-sized Spark-Hadoop Cluster in AWS EC2 for a US Based Developer of Data Storage Hardware and Software Products - + Configured the environment to handle design challenges like memory issues while generating such a large amount of data.+ Generated 5TB of Parquet data on the Hadoop cluster using TPCDS’s dsgen on Spark.+ Utilized Impala Kit Queries to query this data and find the execution times.+ Consolidated and compared results with those published by Cloudera for a similar environment.+ Analyzed the feasibility of replacing S3 as storage instead of using HDFS for this cluster.Hadoop Benchmark testing on Shared storage vs Local storage for a US Based Developer of Data Storage Hardware and Software Products - + Performed TestDFSIO and Teragen-Terasort-Teravalidate tests on a Hadoop cluster with shared and local storages.+ Compared latency and throughput of these tests for varying data and block-sizes.