Project Description To create a production data-lake that can handle transactional processing operations using Hadoop Eco System both streaming and batch processing .Portion of RDBMS tasks are to be migrated to Hadoop for faster & amp; efficient in-memory computations. This project has been a part of ‘Big Data as a Service’ initiative in order to send appropriate alerts to customers.Responsibilities: Analyzing the existing data flow to the warehouses and taking the similar approach to migrate the data into MongoDB. Created Partitioning, Bucketing, and Map Side Join, Parallel execution for optimizing the hive queries decreased the time of execution from hours to minutes. Involved in gathering requirements from client and estimating time line for developing complex queries using Kafka & NIFI streaming techniques.. Worked with cloud provisioning team on a capacity planning and sizing of the nodes (Master and Slave) for an Hortonworks (HDP) Platform Worked with Amazon EMR to process data directly in S3 when we want to copy data from S3 to the Hadoop Distributed File System (HDFS) on your Amazon EMR cluster by setting up the Spark Core for analysis work. Exposure on Spark Architecture and how RDD’s work internally by involving and processing the data from Local files, HDFS and RDBMS sources by creating RDD and optimizing for performance. Involved in data pipeline using Pig, Sqoop to ingest cargo data and customer histories into HDFS for analysis. Worked on importing data from My SQL DB to HDFS and vice-versa using Sqoop to configure Hive

 Involved in requirement gathering with the business stakeholders. Defining Job flows and dependencies Written Unix shell & Python scripts to integrate all the job flows in Autosys scheduler Experience with continuous integration build tools like BitBucket Created virtual environment using conda packages Installed and maintain the python package as per the business needs Created Jyputer Hub for deploying solution to business users for their analytics. Involved in Unit testing, System integration testing and provided support for UAT testing. Involved in deployments in Pre-prod & Production activities. Responsible for ingesting data into HDFS and HBASE from different data sources using Couldera. Developed data ingestion scripts using Sqoop to import data from Netezza ,Oracle & KDBInvolved in doing IMMUTA setup for the data science platform Written shell scripts to invoke Spark,Python ,MapReduce programs, Hive scripts, Sqoop scripts. Worked on performance Tuning and resource management techniques.

 Involved in requirement gathering with the business stakeholders. Defining Job flows and dependencies Written Unix shell scripts to integrate all the job flows in BMC Control-m scheduler Experience with continuous integration build tools like Jenkins ,GITHUB Involved in Unit testing, System integration testing and provided support for UAT testing. Involved in deployments in Pre-prod & Production activities. Responsible for ingesting data into HDFS and HBASE from different data sources using CLOUDERA. Developed data ingestion scripts using Sqoop. Migration of DATA from hive tables to AWS via S3 for data analytics. Handled initial loads and delta during the ingestion process Written shell scripts to invoke Spark,Python , Hive ,IMPALA, Sqoop scripts. Worked on performance Tuning. Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala. Experience in streaming the data using Spark. Import data using Sqoop to load data from RDBMS (Oracle, Teradata,sybase) to HDFS on regular basis. Implemented Daily Oozie jobs that automate parallel tasks of loading the data into HDFS and pre-processing with Pig using Oozie co-coordinator jobs. Created Hive tables and wrote queries to fetch the data from HDFS. Prepared Test Plans and performed regression testing in the Hadoop cluster. Involved in automating the test flows and integrating them in the continuous integration tools like Jenkins Participated in code review sessions.

 Involved in requirement gathering with the business stakeholders. Defining Job flows and dependencies Written Unix shell scripts to integrate all the job flows Experience with continuous integration build tools like Jenkins Involved in Unit testing, System integration testing and provided support for UAT testing. Involved in deployments in Pre-prod & Production. Responsible for ingesting data into HDFS and HBASE from different data sources using MAPR cluster. Developed data ingestion scripts using Sqoop. Handled initial loads and delta during the ingestion process Wrote shell scripts to invoke MapReduce programs, Hive scripts, Sqoop scripts etc. Worked on performance Tuning. Import data using Sqoop to load data from RDBMS (Teradata) to HDFS on regular basis. Implemented Daily Oozie jobs that automate parallel tasks of loading the data into HDFS and pre-processing with Pig using Oozie co-coordinator jobs. Created Hive tables and wrote queries to fetch the data from HDFS. PreparedTest Plans and performed regression testing in the Hadoop cluster. Involved in automating the test flows and integrating them in the continuous integration tools like Jenkins Participated in code review sessions.