Created Sqoop job to bring the data from Oracle to HDFS and created external Hive tables in Hive.Used Kafka consumer’s API in Scala for consuming data from Kafka topics.Knowledge on PySpark and used Hive to analyze sensor data and cluster users based on their behavior in the events.Created External Tables in Hive and saved in ORC file format.Ingested data from RDBMS and performed data transformations, and then export the transformed data to Cassandra as per the business requirement.Built data pipeline using Pig to store onto HDFS.Worked on HiveQL for data analysis for importing the structured data to specific tables for reporting.Ingest data from upstream systems using NiFi into Hive. Create base line tables for trending in HBase using Spark batch process in Scala.Designed and developed Microservices business components using Spring Boot.Implemented spring applications with Spring Boot and DAO design pattern for maximum abstraction of the application and code reusability. Implemented the application using Spring Boot Framework and handled the security using spring security.Worked Big data processing of clinical and non-clinical data using MapReduce. Implemented POC to migrate MapReduce jobs into Spark RDD transformations using Scala. Scheduled MapReduces jobs in production environment using Oozie scheduler. Improved the Performance by tuning of Hive and MapReduce.Developed Spark Applications by using Scala, Java and implemented Apache Spark data Processing Project to handle data from various RDBMS and streaming sources.Wrote Python scripts to parse XML documents and load the data in database. Experience in working with Hive to create Value Added Procedures. Also wrote Hive UDF to make the function reusable for different models.Loaded the dataset into Hive for ETL (Extract, Transfer and Load) operation.Implemented Kafka model which pulls the latest records into Hive external tables.

Developed micro-services using Python scripts in Spark Data Frame API’s for the semantic layer.Involved in complete Implementation lifecycle, specialized in writing custom MapReduce, Pig and Hive. Have been using NiFi for transferring data from source to destination and Responsible for handling batch as well as Real-time Spark jobs through NiFi.Developed a data pipeline using Kafka, Spark and Hive to ingest, transform and analyzing data.Used Scala to convert Hive/SQL queries into RDD transformations in Apache Spark. Built the complete data ingestion pipeline using NiFi which POST’s flow file through invoke HTTP processor to our Micro services hosted inside the Docker containers.Involved in creating Hive tables, loading with data and writing Hive queries which will run internally in MapReduce way.Designed and implemented MapReduce based large-scale parallel relation-learning system.Processed Multiple Data sources input to same Reducer using Generic Writable and Multi Input format. Performed data profiling and transformation on the raw data using Pig and Python.Created Hive Generic UDF's to process business logic that varies based on policy. Moved Relational Data base data using Sqoop into Hive Dynamic partition tables using staging tables. Monitoring Cluster using Cloudera manager.Develop predictive analytic using Apache Spark Scala APIs. Implemented MapReduce counters to gather metrics of good records and bad records.Built data governance processes, procedures, and control for Data Platform using Nifi. Creating real time data streaming solutions and batch style large scale distributed computing applications using Apache Spark, Spark Streaming, Kafka and Flume. Used Oozie to orchestrate the MapReduce jobs that extract the data on a timely manner.Responsible for importing real time data to pull the data from sources to Kafka clusters. Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.

Implement shell scripts to load data from Teradata to hadoop by using spark submit and tech connectors.Used Oozie Scheduler system to automate the pipeline workflow and orchestrate the MapReduce Jobs that extract the data in a timely manner. Responsible for loading data from UNIX file system to HDFS. Created spark streaming jobs which collect data from source adobe data in Hadoop location Push streaming data from Hadoop to Akka collector and to Kafka topic and process data using Scala scripts in spark. Implemented Spark using Scala, Python and utilizing Data frames and Spark SQL API for faster processing of data.Comprehensive experience in using Node.js and Spring Boot implementing RESTful web service. Hosted and consumed RESTful Web services using JAX-RS API with spring Boot.Involved in creating Hive managed / external tables while maintaining raw files integrity and analyzed data using Hive queries. Used Oozie and Oozie coordinators to deploy end to end data processing pipelines and scheduling workflows. Worked on CI/CD pipeline, integrating code changes to GitHub repository and build using Jenkins.Configured Kafka for efficiently collecting, aggregating and moving large amounts of click stream data from many different sources to HDFS. Monitored workload, job performance and capacity planning using Cloudera Manager. Experience in working with Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the Hive queries. Built real time pipeline for streaming data using Kafka and Spark Streaming.

Understanding and analyzing the requirements.Implemented server side programs by using Servlets and JSP.Designed, developed and validated User Interface using HTML, Java Script, XML and CSS.Implemented MVC using Struts Framework.Handled the database access by implementing Controller Servlet.Implemented PL/SQL stored procedures and triggers.Used JDBC prepared statements to call from Servlets for database access.Designed and documented of the stored procedures.Widely used HTML for web based design.Involved in Unit testing for various components.Worked on database interaction layer for insertions, updating and retrieval operations of data from oracle database by writing stored procedures.Involved in development for simulator which is being used for controllers to simulate real time scenarios using C / C++ programming. Used Spring Framework for Dependency Injection and integrated with Hibernate.Involved in writing JUnit Test Cases