Raj Kumar Email & Phone Number

Indianapolis, Indiana, United States

• Used Agile methodology in developing the application, which included iterative application development, weekly Sprints, stand up meetings and customer reporting backlogs.• Development of efficient pig and hive scripts with joins on datasets using various techniques.• Automating the ETL tasks and data work flows for the data pipeline of the ingest process through UC4 scheduling tool.• Solved performance issues in Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.• Created data sharing between two snowflake accounts.• Wrote ETL jobs to read from web APIs using REST and HTTP calls and loaded into HDFS using java and Talend.• Created internal and external stage and transformed data during load.• Monitoring resources and Applications using AWS Cloud Watch, including creating alarms to monitor metrics such as EBS, EC2, ELB, RDS, S3, SNS and configured notifications for the alarms generated based on events defined.• Analyze and develop programs by considering the extract logic and the data load type using Hadoop ingest processes using relevant tools such as Sqoop, Spark, Scala, Kafka, Unix shell scripts and others• Assist with the analysis of data used for the tableau reports and creation of dashboards. • Design and implement large scale distributed solutions in AWS.• Optimized Map Reduce Jobs to use HDFS efficiently by using various compression mechanisms.• Used ORC and Parquet file formats in Hive.• Writing code and creating hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.• Created and managed cloud VMs with AWS EC2 Command line clients and AWS management console.• Migrated on premise database structure to Confidential Redshift data warehouse. Worked on AWS Data Pipeline to configure data loads from S3 into Redshift.

Pennsylvania, United States

• Developed HIVE UDFs to incorporate external business logic into Hive script and Developed join data set scripts using HIVE join operations.• Enabling monitoring and azure log analytics to alert support team on usage and stats of the daily runs• Implement IOT streaming with Databricks Delta tables and Delta lake to enable ACID transaction logging• Exposed transformed data in Azure Spark Databricks platform to parquet formats for efficient data storage• Used Delta Lake as it is an open-source data storage layer which delivers reliability to data lakes.• Developed and designed data integration and migration solutions in Azure.• Migrate data into RV Data Pipeline using Databricks, Spark SQL and Scala.• Used Databricks for encrypting data using server-side encryption.• Involved in creating HDInsight cluster in Microsoft Azure Portal also created Events hub and Azure SQL Databases.• Configured Spark streaming to get ongoing information from the Kafka and store the stream information to HDFS.• Took proof of concept projects ideas from business, lead, developed and created production pipelines that deliver business value using Azure Data Factory• Worked with PowerShell and UNIX scripts for file transfer, emailing and other file related tasks.• Delivered de normalized data for Power BI consumers for modeling and visualization from the produced layer in Data lake• Extracted and updated the data into HDFS using Sqoop import and export.• Create custom logging framework for ELT pipeline logging using Append variables in Data factory• Developed various Oracle SQL scripts, PL/SQL packages, procedures, functions, and java code for data• Strong Knowledge on architecture and components of Tealeaf, and efficient in working with Spark Core, SparkSQL. Designed and developed RDD Seeds using Scala and Cascading. Streaming data to Spark streaming using Kafka.

Franklin, Tennessee, United States

• Developed SSRS reports, and SSIS packages to Extract, Transform, and Load data from various source systems• Implementing and Managing ETL solutions and automating operational processes.• Was responsible for ETL and data validation using SQL Server Integration Services.• Measured Efficiency of the Hadoop/Hive environment ensuring SLA is met• Optimized the Tensor Flow Model for efficiency• Created Entity Relationship Diagrams (ERD), Functional diagrams, Data flow diagrams and enforced referential integrity constraints, and created logical and physical models using Erwin.• Defined and deployed monitoring, metrics, and logging systems on AWS.• and Spark SQL for ETL jobs and using the right technology for the job to get done.• Compiled data from various sources to perform complex analysis for actionable results• Strong understanding of AWS components such as EC2 and S3• Managed security groups on AWS, focusing on high availability, fault-tolerance, and auto-scaling using Terraform templates. Along with Continuous Integration and Continuous Deployment with AWS Lambda and AWS code pipeline.• Defined facts, and dimensions and designed the data marts using Ralph Kimball's Dimensional Data Mart modeling methodology using Erwin.• Worked on Big data on AWS cloud services i.e. EC2, S3, EMR, and DynamoDB• Created ad hoc queries and reports to support business decisions SQL Server Reporting Services (SSRS).• Analyze the existing application programs and tune SQL queries using an execution plan, query analyzer, SQL Profiler and database engine tuning advisor to enhance performance.• Optimizing and tuning the Redshift environment, enabling queries to perform up to 100x faster for Tableau and SAS Visual Analytics.

India

• Developed PIG scripts for the analysis of semi-structured data.• Worked on Oozie workflow engine for job scheduling.• Worked with Avro Data Serialization system to work with JSON data formats.• Hive Context, with transformations and actions (map, flat Map, filter, reduce, reduce by Key).• Migrated ETL jobs to Pig scripts to do Transformations, even joins, and some pre-aggregations before storing the data onto HDFS.• Performed tuning of SQL queries and Stored Procedures for speedy extraction of data to resolve and troubleshoot issues in the OLTP environment.• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.• Worked on different file formats like Sequence files, XML files, and Map files using MapReduce Programs.• Developed PIG UDF for manipulating the data according to Business Requirements and also worked on developing custom PIG Loaders.• Worked on various performance optimizations like using distributed cache for small datasets, partition and bucketing in Hive, doing map side joins, etc.• Performed pre-processing on a dataset prior to training, including standardization, and normalization.• Worked with heterogeneous source to Extract data from Oracle database, XML, and flat files and loaded to a relational Oracle warehouse.• Evaluated model accuracy by dividing data into training and test datasets and computing metrics using evaluators.

India

• Normalized tables, established joins, and created indexes wherever necessary utilizing profiler and execution plans.• Utilized dimensional data modeling techniques and storyboarding ETL processes.• Hands-on experience in analyzing the data and writing Custom MySQL queries for better performance, joining the tables and selecting the required data to run the reports.• Restricted data for Users using row level security and use filters.• Developed ETL procedures to ensure conformity, compliance with standards and lack of redundancy, translated business rules and functionality requirements into ETL procedures using Informatica Power Center.• Utilized technology such as MySQL and Excel PowerPivot to query test data and customize end-user requests.• Performed Unit Testing and tuned the Informatica mappings for better performance.• Re-engineer existing Informatica ETL process to improve performance and maintainability.