Data Pipeline Development & Migration: Designed and implemented end-to-end data pipelines using GCP (Dataflow/DataProc, Cloud Functions/Cloud Composer) to ingest and transform data into BigQuery and Cloud SQL. This involved various ingestion patterns (full load, incremental load, CDC). While specific volume isn't stated, the implication is significant data volume given the use of these tools and the scale of the company. The impact is improved data accessibility and processing efficiency.CI/CD Pipeline Implementation: Created cloud-based CI/CD pipelines using Terraform, streamlining deployment and improving the reliability of the data infrastructure. Quantifiable metrics here would be deployment frequency and time reduction

• Identified anomalies in the time-series dataset using GCP components. • Implemented DataProc-Spark pipelines using files placed in cloud storage to ingest and transform data into BigQuery datasets and Cloud SQL.• Developed DQ framework for SCA-V Products.• Implemented real-time POC streams for processing upstream auto quality checks.• Develop/Automated DQ framework for TCX.

•Spark and Scala developing for streaming applications; containerized deployment of continuous pipelines for the operational and analytical teams.•Built a real-time streaming application E-commerce global reservations for Ford electrical vehicles (Mustang Mach-e) from front-end apps through Kafka and store the stream data to HDFS and processed using Spark Structured Streaming.•Implemented Spark in-memory (updates & delete’s) data ingestions technique for CCPA project which is one of the large datasets for third-party sources.•Performance & Tuning SQL queries using distributed cache for large datasets and also Partition, Bucketing, Map Side joins, and ETL Jobs related scripts in a Hive.

•Assists in delivering the information standard to the Business Customers and developing a robust maintenance process for the solution with existing IT and Enterprise Architecture strategies.•Cleans, Standardize and enrich Ford’s Global Customer Data using Hadoop big data technologies such as Spark, Hive, HBase, Alteryx, and SAS.•Developed and support Enterprise Data Products such as Data Quality, Monitoring & Profiling tools with managing Data Governance principles and handled ≃10PB of large datasets using Apache Spark in-memory & parallel processing capabilities, continuously evaluating performance scale across Hadoop Data Lake.•Analyzed & identified data failures gaps for Data Landings and Processing I built Patterns Recognition based on existing data injections models and expecting Outlier threshold values for missing data by using Spark MLLib (Clustering Algorithms such as K-distance neighborhood and Euclidean distance in Scala).•Provided visibility for monitoring operational data by deploying a predictive model to detect anomalies based on record counts, cardinality and other data quality issues.•Develop regular expressions (REGEX) & enriching reference libraries (LIB) to identify patterns in data and manage the ad-hoc priority loads, working with cross-functional and cross-domain stakeholders in creating priority lists for landing diverse data sources.•Worked on (Graph API) for ETL data injections and creating reports which are useful for business stakeholders on events like ads, campaigns & marketing from web analytical tools like Adobe Click Stream Analytics & Facebook Business Manager.

•Created end to end Data pipeline for healthcare claims-related data and streams it into a data repository in a Hadoop Data-lake environment.•Developed Data Products using Apache Spark (Scala) for doing predictive models for reducing the underlying behavioral health disorders based on healthcare marketers.•Automated weekly campaigns for On-track providers using there Salesforce Member ID’s & Marketo leads with Hadoop environment using Spark API's data-sets and storing data into a hive for data-analysis.•Created Algorithm for helping data sharing, data reuse, data quality and data standards based on Health Client HIPAA data. •Designed and Implemented to consume PHI raw user events from Kafka and generate meaningful metrics for analysis. •Developed process and performance metrics to extract data and performed quality control to support Data Science teams.•Supported the Extract Transform Load (ETL) processes for the existing data warehouse and handled process failures for ad-hoc data requests from various business and technical teams.•Created Looker monthly on-track reports for data visualization to Stakeholders.

•Collected the historical data of stocks, stored this dataset into HBase and found the top five highest and lowest volatile stocks by running Map Reduce job on Hadoop Cluster. •Handled importing of data from various data sources, performed transformations using Hive, PIG, and loaded data into HDFS. •Experienced in running Hadoop Map-reduce jobs to process terabytes of data.•Monitoring and analysis of the Map Reduce job executions on cluster at task level.•Experience in Importing and exporting data into HDFS and Hive using Sqoop.•Changes to the configuration properties of the cluster based on volume of the data being processed and performance of the cluster. •Load and transform large sets of structured, semi structured and unstructured data. •Performed multiple Map side jobs in PIG and Hive for data cleaning and pre-processing.•Responsible for managing data coming from different sources. •Worked on NoSQL (HBase) for support enterprise production and loading data into HBASE using HIVE and SQOOP.•Creating Hive tables and working on them using Hive QL.•Involved in creating Hive tables, loading with data and writing hive queries, which will run internally in map and reduce way. •Involved in creating tables, partitioning, bucketing of table in Hive. •Analysis of streaming twitter data, we used Twitter API to gather Tweets based on Keyword and Data transformation was done using Spark.•Visualized and processed data using spark streaming and generated plots and graphs to display reports.•Good understanding and related experience with Hadoop stack-internals, Hive, Pig and Map/Reduce.

This project is to support data analytics group for reporting tools and dashboard delivery.I am responsible for collecting and analyzing raw data and creating interactive report dashboards.To comparatively process raw tweets data and perform various analytics depend upon the requirement and to analyze the number of hits to a website or a keyword during a period.