Vineeth Kumar Email & Phone Number

Miami, Florida, United States

• Streamlined the Data Ingestion from different sources including Streaming, Batch, and Databases as part of AWS Data Lake implementation with the help of Spark on AWS Databricks, Delta Lake. • Contributed to multiple ETL batch and Stream processing Spark jobs to transform and load the data to different data zones in Pearson Data Lake. • Implemented the workflows using the Apache Airflow, and AWS Databricks Workflows to orchestrate job execution • Integration of AWS Databricks Unity Data Catalog as part of Datalake Governance. • Utilized Delta Lake, a reliable and scalable data lake solution on Azure Databricks, to ensure data integrity, ACID transactions, and versioning for large-scale data processing and analytics.• Developed and orchestrated complex data workflows using Databricks Jobs and Workflows, enabling automated data processing, scheduling, and monitoring.• Integrated AWS Databricks with AWS Data Lake Storage to efficiently store and manage structured, semi-structured, and unstructured data at scale, ensuring data governance and compliance.• Implemented data ingestion pipelines on AWS Databricks, using AWS S3, Apache Kafka, or other relevant technologies, for real-time and batch data streaming scenarios Worked on building the Prediction stream where we predict the streaming data from more than 100 deployed models on MLFlow using PySpark.• Discovery and implementation of generic rest services for the data lake starting from config service, metadata service, and many reusable client components with Python Django.• In-depth understanding of Spark Architecture including Spark Core, Spark SQL, and Data Frames.• Written transformations and actions on data frames, used Spark SQL on data frames to access hive tables into Spark for faster processing of data.• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Python.

St Louis, Missouri, United States

• Built a framework for the Customer Master Data Platform to Profile the millions of customers and the framework enables to derivation of new custom attributes with different clients (Including Spark, SQL, and Python)• Deployed Customer Master Data platform pipeline on AWS, all orchestrations done with lambda, step functions, and AWS cloud watch. Built the rest APIs using Python Django on top of our framework.• Designed and Developed a Customer Unified User Profile framework (Discovering Audiences) to identify profiled users and Provide personalized offers. Deployed on EMR with 12 node cluster by caching terabytes of data. • Worked on URL tracking with spark streaming and Predicted real-time results.• Extensively Used Apache Spark, and Apache Flink for building real-time Streaming and batch jobs to process 10^6 events per minute. Contributed to multiple Streaming and structured streaming jobs which take care of ingesting the data from multiple channels to the platform, real-time batch Processing from streaming data, Streaming Aggregations of data, Event Analyzers, and Influx Sink.• Designed and Developed data integration/engineering workflows on big data technologies and platforms (Hadoop, Spark, Hive). • Handled importing of data from various data sources, performed transformations using Spark, and loaded data into S3.• Processed S3 data and created external tables using Hive and developed scripts to ingest and repair tables that can be reused across the project.• Developed dataflows and processes for the Data processing using SQL (SparkSQL& Dataframes)• Developed spark programs using Python APIs to compare the performance of spark with HIVE and SQL and generated reports monthly and daily basis• Experience with AWS cloud services like EMR, EC2, S3, and Athena.

New York, United States

Worked extensively in AWS to Azure Cloud Migration on behalf of a Microsoft consultant.• Migrated Click Stream pipeline from scratch to flow the data from transaction databases to data lakes and Dataware houses with the help of open-source selenium, Kafka, and spark stream processing • Migrated Terrabytes of data from AWS to Azure by creating migration data pipelines in Azure Data Factory• Had set up More than 150 Big data pipelines in Azure HD insights and orchestrated them through Airflow• Experience in troubleshooting issues and failed jobs in the Hadoop cluster.• SQL querying and performance tuning, creating backup tables.• Implemented partitioning, dynamic partitioning, and bucketing in the hive• Handled importing of data from various data sources, performed transformations using Spark and Hive in both AWS EMR and Azure HD Insights, and loaded data into S3.• Created Hive Queries to process large sets of structured, semi-structured, and unstructured data and store them in Managed and External tables• Experience analyzing data solving problems and troubleshooting to provide solutions. Identifying data requirements, performing data mapping, and testing data integration.• Worked on data pre-processing and cleaning the data to perform feature engineering and performed data imputation techniques for the missing values in the dataset using Python.• Used Amazon AWSEMR and EC2 for cloud big data processing.

Hyderabad, Telangana, India

• Built multiple rest endpoints for Finance Clients with Python Django framework that connects to MongoDB to fetch the required data. • Written multiple Aggregated queries on top of MongoDB on Giga bytes of data.• Built extract, transform, and load(ETL) process to migrate data from multiple types of data sources to destination Mongo DB.• Built CI & CD deployment with Jenkins and Generated code quality report with Sonar Qube• Extensively used Python language to develop web applications with MVC patterns.• Identified issues with the database, and performed extensive optimizations to resolve them.• Explored different Python frameworks including Flask, fast-API, and Django to build microservice-based deployments• Automated repetitive deployment processes using UNIX shell scripts and Docker containerization for multiple web applications.• Handled errors using Exception Handling extensively for the ease of debugging and displaying the error messages in the application.