Rajesh M. Email & Phone Number

Oklahoma City, Oklahoma, US

• Design and developed the streaming framework which ingest the data from sources like Kafka, S3 to AWS S3, Kafka, GreenPlum.
• Developed the pipeline to ingest the data from MySql to AWS S3 using Spark & deployed to Kubernetes cluster using Helm Charts.
• Developed the helm chart for a Spark job to deploy multiple jobs in Kubernetes cluster.
• Design and developed the Spark data optimization pipeline which deletes the data of a given AWS S3 delta lake dataset and performs optimize & vacuum using PySpark.
• Worked on Kubernetes deployments of Spark jobs using Helm charts.
• Implemented jobs to ingest streaming data from Kafka to Greenplum database using GPSS (Greenplum Streaming Server).

Bentonville, Arkansas, US

• Developed the ETL process using PySpark and converted the alteryx workflows to PySpark.
• Migrated the workflows from Cloud break cluster to Data Proc cluster.
• As part of GCP migration, converted the complex hive scripts to Spark Scala.
• Written Spark scripts in order to load the target data into Hudi tables.
• Applied Spark performance techniques in order to optimize and improve the query run time.
• Worked on building the pipelines in Atomic to load the data from raw zone to GCS stage bucket, performed various transformations and loaded the data into Big Query tables.

Mooresville, NC, US

• Developed a framework to auto trigger the oozie jobs based on the status using oozie web service, hbase and Scala.
• Developed python script to convert the JSON files to csv before it load into HDFS.
• Implemented the check point mechanism in order to rerun the job from the failure step in Spark Python.
• Optimized Hive queries using Bucketing.
• Converted complex Hive queries into Pyspark.
• Developed complex requirements in dataframe’s using Pyspark and pushed into Hbase for dashboard.

Rosemead, CA, US

• Worked on analyzing Hadoop stack and different Big data tools including Pig and Hive, Hbase database and Sqoop.
• Used Talend ETL tool to perform data processing from source to Stage and Stage to Core.
• Developed an automated ingestion framework using Talend tool to load data from BW SAP to HDFS.
• Developed data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.
• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Worked with different data sources like Avro data files, XML files, JSON files, SQL server and Oracle to load data into Hive tables.

• Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO and Parquet to HDFS cluster with compressed for optimization.
• Worked on ingesting data from RDBMS sources like - Oracle, SQL Server and Teradata into HDFS using Sqoop.
• Loaded all data-sets into Hive from Source CSV files using Spark.
• Built a Ingestion Framework that would ingest the files from SFTP to HDFS using Apache NIFI.
• Configured Hive and written Hive UDF's and UDAF's Also, created Static and Dynamic with bucketing as required.
• Managing and scheduling Jobs on a Hadoop cluster using Oozie.

Bedford, MA, US

• Installed & configured multi-node Hadoop Cluster and performed troubleshooting and monitoring of Hadoop Cluster.
• Managing, defining and scheduling Jobs on a Hadoop cluster.
• Wrote Hive queries for ad-hoc reporting and ETL.
• Worked with different file formats such as Text, Sequence files, Avro, ORC and Parquette.
• Worked on NOSQL database Hbase.
• Optimizing Hadoop MapReduce code, Hive and Pig scripts for better scalability, reliability and performance.

Masters

Bachelor'S Degree, Computer Science