• Actively involved in designing Hadoop ecosystem pipeline.• Working on Big Data infrastructure for batch processing as well as real-time processing. Responsible for building scalable distributed data solutions using Hadoop.• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.• Involved in designing Kafka for multi data center cluster and monitoring it.• Responsible for importing real time data to pull the data from sources to Kafka clusters.• Responsible for design and development of Spark SQL Scripts based on Functional Specifications. • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, and Scala. • Developed Spark Applications by using Scala, Java, Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.• Used Spark SQL on data frames to access hive tables into spark for faster processing of data. • Working knowledge of Spark RDD, Data Frame API, Data set API, Data Source API, Spark SQL and Spark Streaming.• Used Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames.• Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications. • Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS. • Worked on the large-scale Hadoop YARN cluster for distributed data processing and analysis using Spark, Hive.• Involved in creating data-lake by extracting customer's data from various data sources to HDFS which include data from Excel, databases, and log data from servers.• Created various hive external tables, staging tables and joined the tables as per the requirement. Implemented static Partitioning, Dynamic partitioning and Bucketing in Hive using internal and external table.

• Involved in implementation of Hadoop Cluster and Hive for Development and Test Environment.• Worked on analyzing Hadoop Cluster and different big data analytic tools including Pig, Hive and MongoDB. Extracted files from MongoDB through Sqoop and placed in HDFS for processed.• Implemented Spark Scripts using Scala, Spark SQL to access hive tables into spark for faster processing of data. • Loaded the data into Spark RDD and do in memory data Computation to generate the faster Output response.• Created the spark SQL context to load the data from hive tables into RDD’S for performing complex queries and analytics on the data present in data lake.• Developed a Nifi Workflow to pick up the data from SFTP server and send that to Kafka broker. Loaded D-Stream data into Spark RDD and did in-memory data computation to generate output response. • Spark Streaming collects data from Kafka in near-real-time and performs necessary transformations and aggregation to build the common learner data model and stores the data in NoSQL store (MongoDB).• Reading the log files using Elastic search Logstash and alerting users on the issue and also saving the alert details to MongoDB for analyzations.• Wrote queries in MongoDB to generate reports to display in the dash board.• Worked on MongoDB database concepts such as locking, transactions, indexes, sharding, replication and schema design.• Used MongoDB to store Bigdata and applied aggregation Match, Sort and Group operation in MongoDB.• Hands on experience in writing custom UDF’s, custom input and output formats and created Hive Tables, loaded values and generated adhoc-reports using the table data.• Showcased strong understanding on Hadoop architecture including HDFS, MapReduce, Hive, Pig, Sqoop and Oozie.• Worked extensively on Hive to create, alter and drop tables and involved in writing hive queries.

• Importing and exporting data into HDFS, Pig, Hive and HBase using SQOOP.• Experience in installing Hadoop cluster using different distributions of Cloudera distribution.• Responsible for analyzing large data sets and derive customer usage patterns by developing new MapReduce programs.• Written MapReduce code to parse the data from various sources and storing parsed data into Hbase and Hive.• Created HBase tables to store different formats of data as a backend for user portals. • Developed Kafka producer and consumers, HBase clients, Apache Spark and Hadoop MapReduce jobs along with components on HDFS, Hive. • Successfully migrated Legacy application to Big Data application using Hive/Pig/HBase in Production level.• Load and transform large sets of structured, semi structured, and unstructured data that includes Avro, sequence files and XML files.• Involved in gathering the requirements, designing, development and testing. • Implemented helper classes that access HBase directly from java using Java API to perform CRUD operations. • Handled different time series data using HBase to perform store data and perform analytics based on time to improve queries retrieval time.• Integrated Map Reduce with HBase to import bulk amount of data into HBase using Map Reduce Programs.• Developed simple and complex MapReduce programs in Java for Data Analysis.

• Involved in loading data from UNIX file system to HDFS. • Worked extensively on Hive and written Hive UDFs. • Importing and exporting data into HDFS and Hive using Sqoop.• Handled importing of data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS.• Analyzed the data by performing Hive queries and running Pig scripts to know user behavior.• Exported the patterns analyzed back into Teradata using Sqoop. • Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.• Using Sqoop to load data from DB2 into HBASE environment. • Inserted Overwriting the HIVE data with HBase data daily to get fresh data every day. • All the bash scripts are scheduled using Resource Manager Scheduler. • Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, and visit duration. • Finding the solutions to the bottlenecks in high latency hive queries through analyzing log messages. • Developed Oozie Workflows for daily incremental loads, which gets data from Teradata and then imported into hive tables. • Extensively used MapReduce Design Patterns to solve complex MapReduce programs.

• Participated in major phases of software development cycle with requirement gathering, Unit testing, development, and analysis and design phases using Agile/SCRUM methodologies. • Involved in gathering and analyzing system requirements.• Used Multithreading and exceptional handling in the development of applications.• Developed application is based on the MVC-II Architecture using Apache Struts framework.• Migrated some modules from VB6.0 to java.• Wrote C++ programs for end of day operations (EOD) to close or roll the orders.• Designed and developed user interface screens using HTML, JQuery and JSP.• Created and maintained the configuration of the Application Framework.• Eclipse used as Java IDE tool for creating Action classes and XML files.• Implemented the application with Spring Framework for implementing Dependency Injection and provide abstraction between presentation layer and persistence layer.• Developed multiple batch jobs using Spring Batch to import files of different formats like XML, CVS etc. • Involved in development of application using Rule Engine(Drools).• Used Rule Engines in applications to replace and manage some of the business logic.• Wrote business rules using Drools and business logic processing customs declarations. • Monitored Logs files and troubleshooting environment variable in Linux boxes.• Involved in maintenance of the application.