Venu Babu B Email and Phone Number

• Designed, developed, and maintained data pipelines for retail use cases, processing vast datasets for forecasting of items, club-level ranking of items, and sourcing.• Leveraged Google Cloud Platform (GCP) services, including Google Dataproc, Google Cloud Storage (GCS), and BigQuery to enhance data storage, analysis, and visualization.• Utilized Apache Spark with Scala and Python to build high-performance, distributed data processing solutions, optimizing data processing times by 40%.• Expertise in working with various databases, including Hive, Oracle, PostgreSQL, and DB2, optimizing data retrieval and storage strategies (ORC, Parquet).• Implemented Kafka for real-time data streaming, enabling near-instantaneous data processing for time-sensitive applications.• Integrated Azure services for hybrid data solutions, enhancing data accessibility and availability.• Assisted in the evaluation and selection of data engineering tools, frameworks, and technologies to keep the data stack up-to-date and aligned with industry best practices.• Create SQL, PL/SQL scripts for sourcing data, including creating tables, Materialized views, stored procedures, and loading data into the tables.

Performed end-to-end Architecture and implementation assessment of various AWS serviceslike Amazon EMR, Redshift, S3, Athena, Glue, and Kinesis.Involved in code migration of quality monitoring tool from AWS EC2 to AWS Lambda andbuilt logical datasets to administer quality monitoring on snowflake warehouses.Created ETL jobs on AWS glue to load vendor data from different sources, transformationsinvolving data cleaning, data imputation and data mapping and storing the results into S3 buckets.The stored data was later queried using AWS Athena.Experienced in performance tuning of Spark Applications for setting right Batch Interval time,correct level of Parallelism and memory tuning.Used AWS data pipeline for Data Extraction, Transformation and Loading from homogeneousor heterogeneous data sources and built various graphs for business decision-making using Pythonmat plot library

Involved in file movements between HDFS and AWS S3 and extensively worked with S3bucket in AWS and converted all Hadoop jobs to run in EMR by configuring the cluster according tothe data size.Worked on data pipeline creation to convert incoming data to a common format, prepare datafor analysis and visualization, Migrate between databases, share data processing logic across webapps, batch jobs, and APIs, Consume large XML, CSV, and fixed-width files and created datapipelines in Kafka to replace batch jobs with real-time data.Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performsnecessary Transformations and Aggregations on the fly to build the common learner data model andpersistence the data in HDFS.Extensively used Stash Git-Bucket for Code Control and Worked on AWS Components suchas Airflow, Elastic Map Reduce (EMR), Athena and Snowflake.

Developed data pipeline using Flume to ingest data and customer histories into HDFS foranalysis.Involved in moving all log files generated from various sources to HDFS for furtherprocessing through Kafka.

Extract Real-time feed using Spark Streaming and convert it to RDD and process data in theform of Data Frame and save the data in HDFS.Worked on analyzing Hadoop clusters using different big data analytic tools including HBasedatabase and Sqoop.

Coded different deployment descriptors using XML. Generated Jar files are deployed on ApacheTomcat Server.Implemented Java/J2EE Design patterns like Business Delegate and Data Transfer Object (DTO),Data Access Object.