Project name : Real-time data ingestion and ETL process of Iot edge devices that sends engine data to the Azure cloudLeading the project to design and implement end to end data pipeline solution for ingesting the real time streaming dataset from IoT devices.Enabled the structured streaming solution to pull live message from the Amazon AWS/SQS and ingest into the azure data lake.Handled very large dataset in real time that sends upto 500K unstructured files per day with 20K unique parameters.Applied multiple optimization techniques on the delta tables to improve the performance of the streaming pipeline.Conducted end to end stress/load testing to test the solution for various input loads before productionizing.Designed and implemented the audit and alert mechanism to capture key logs on the data status of each files and visualized results in powerBI.Lead the data engineering team to handle all the IoT data coming from different data sources and provide continous support to the ongoing data pipelines.Collaborate with (internal stakeholders) the upstream and downstream teams to ingest data from various data sources to support data scientist and business analytics needs.Continuously look for improvements within the team and identify the gap within data quality, alert mechanism and data reconciliationComplete various courses from databricks learning path for data engineer using spark scala and spark SQL.

Worked on ingesting complex dataset from various Cummins data sources such as IoT data sources, oracle source, SFTP, Salesforce and SQL.Ability to optimize SQL queries for huge volume of data.Worked on the advance analytics product tool to detect and solve failure modes that cause unexpected downtime performance across mining and XBL engine population and issues leading to catastrophic damage.Owned the weekly data pipeline refresh for business-critical projects over several weeks that enabled the business to get updated data weekly and taken responsibility to migrate the entire product to the production support team.Created and maintaining end to end data engineering pipelines from RAW to REPORTING layer for new data sources in SPARK SCALA to create reports in SQL DB to fullfill the business requirement.Successfully Orchestrated and Scheduled workflow with ~100 Hadoop jobs across platforms e.g. Databricks and HDInsight.Worked as a part of the agile scrum team to achieve the product goal and in release of incremental deliverables to the customers on time. Aided from data engineer perspective to take critical technical decision.

Developed web application interface and code behind using C# and Microsoft SQL server for complex business requirements.Developed a fully automated ERP system to streamline multiple functionalities across the organization and manage the data using Microsoft Azure Storage.Created complex XML, EDI translator and parser using C# web services.

Research, Visualization Lab(May 2015 – Dec 2016)Developed a new geo-visualization algorithm for spatio-temporal data using time curves in two-dimensional space and published the paper in IEEE journal wth title “Spatiotemporal Visualization of Traffic Paths Using Time Curves in Color Space”Improved algorithm efficiency by implementing parallellization of the streaming dataset based on different time components to visualize trends and analyze patterns.

Developed and successfully deployed framework to measure and analyze performance of concurrent queries using Vertica big data analytic database and services, strategy to calculate query efficiency and visualize test performance statistics.Progressively accomplished features from prototype, design, coding, unit and integration testing.Built a comprehensive Looker dashboard that serves as an interactive front-end UI to display the results and to visualize the concurrent query performance using bash script, kafka, Looker, VSQL. Worked using Jenkins to run automated scripts and employed Git plugin for continous integration.

Drafted key performance indicators and automated one stop dashboard to measure various holistic views of campaign performance.Hands-on experience in Teradata, SQL, R, SAS, Tableau, and Python for end to end data mining, business analysis, data visualization and interpretation.Expertise in handling big data and optimizing SQL queries.Led team of 6 members for a new marketing campaign initiative to analyze business requirements.Communicated actionable insights and provided recommendations to help marketers develop strategic decisions.Experience in various machine learning algorithms such as customer profiling and segmentation, clustering techniques, linear and logistic regression model, time series forecasting.Analyzed A/B tests on website, aimed at improving user engagement and consumer acquisition.