 Worked towards designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, HIVE, HBase, Oozie, ZooKeeper, SQOOP, and Spark. Developed Scala scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through SQOOP.  Worked on improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frames, RDD's, Spark YARN.  Responsible for Developing Data Pipeline to load data from sources such as IBM Mainframes and SQL Server using SQOOP along with Kafka and Spark Streaming & Processing Frameworks as per the requirements.  Imported data from Kafka Consumer group into Apache Spark through Spark Streaming APIs.  Worked towards Real-time Streaming of data using Spark with Kafka. Performed Advanced analytics, feature selection/extraction using Apache Spark (Machine Learning & Streaming libraries) in Scala. Worked extensively with importing metadata into Hive using Scala and migrated existing tables and applications to work on Hive. Transferred Data from Legacy Systems to HDFS and HBase using SQOOP. Loading data into HBase using Bulk Load and Non-bulk load. Involved in collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.  Responsible for developing Data Pipeline using SQOOP and Pig to extract data from weblogs and store in HDFS. Gained extensive knowledge and exposure on PySpark and various Spark API’s. Developed POC's using Scala, Spark SQL and MLlib libraries along with Kafka and other tools as per requirement then deployed on the Yarn cluster. Developed POC to configure and Install Apache Hadoop in AWS EC2 System. Further, Casandra Cluster was deployed in Amazon AWS environment with high level of scalability as per requirements.

 Worked on Hadoop environment with MapReduce, Kafka, SQOOP, Oozie, Flume, HBase, Pig, HIVE and Impala on a multi node cluster.  Gathered the business requirements from the business partners and the subject matters experts.  Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.  Supported Map Reduce Programs those are running on the cluster and also wrote MapReduce jobs using Java API.  Installed Oozie workflow engine to run multiple HIVE and Pig jobs.  Gained experience in running Hadoop streaming jobs to process terabytes of XML format data. Importing and exporting data into HDFS and HIVE using SQOOP.  Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.  Worked on Cluster coordination services through ZooKeeper.  Monitored workload, job performance and capacity planning using Cloudera Manager.  Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS. Performed data importing from various sources to HBase and Cassandra cluster using Sqoop.  Used Cassandra (CQL) with Java API's to retrieve data from Cassandra tables.  Used HIVE and created HIVE tables and involved in data loading and writing HIVE UDFs. Gained knowledge on building Apache Spark applications using Java and Scala. Implemented Kafka Custom encoders for custom input format to load data into Kafka Partitions.  Worked on real time streaming the data using Spark with Kafka for faster processing.  Experience in developing and designing POC's using Scala, Spark SQL and MLlib libraries then deployed on the Yarn cluster. Worked on No-SQL databases like HBase, MongoDB for POC purpose in storing images and URIs.  Responsible for implementing POC's to migrate iterative map reduce programs into Spark transformations using Spark and Scala.

 Developed multiple MapReduce jobs in java and Python for data cleaning and preprocessing. Developed the Java client API for node provisioning, load balancing and artifact deployment.  Implemented JMS for asynchronous auditing purposes. Designed and implemented Map reduce jobs to support distributed processing using java, HIVE and Pig.  Maintenance of data importing scripts using HIVE and Map reduce jobs. Wrote MapReduce jobs using Pig Latin.  Developed workflow using Oozie for running MapReduce jobs and HIVE Queries. Worked on loading log data directly into HDFS using Flume.  Involved in loading data from LINUX file system to HDFS.  Experienced in running Hadoop streaming jobs to process terabytes of XML format data.  Written Pig and HIVE jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data. Also have hand on Experience on Pig and HIVE User Define Functions (UDF). Involved in Extracting, loading data from HIVE into RDBMS using SQOOP.  Integrate four square monitoring and production system with Kafka.  Optimized existing reports for performance and generated new reports. Worked with the advanced analytics team to design fraud detection algorithms and then developed MapReduce programs to efficiently run the algorithm on the huge datasets in Python. Extending HIVE and PIG core functionality by using custom User Defined Function's (UDF), and User Defined Aggregating Functions (UDAF) for Hive and Pig using Python.  Developed multiple Map Reduce programs in Python to process large volumes of semi/unstructured data files using different Map Reduce design patterns.  Developed HBase data model on top of HDFS data to perform near real time analytics using Java API.  Used Oracle Java Developer and SQL Navigator as tools for Java and PL/SQL development.

 Developed the application following Agile scrum methodology and project tracking was done using Rally. Developed the project using Spring MVC, JQuery, JSP, HTML, CSS and JavaScript. Involved in designing and coding of Controllers for all modules using Spring MVC framework. Used the spring framework for Inversion of Control (IOC)/Dependency Injection. Responsible for developing the UI pages using HTML, DHTML, CSS, Java Script, JSP, Ajax, JSTL, JSP tag libraries, and custom tags. Used Hibernate ORM framework in the Data Access layer to interact with the Oracle database. Created Hibernate mapping files mapping DAO’s to the tables in the database. Coded Data Objects, Data Access Objects, and Business Objects in the application. Involved in creating JNDI lookup to locate the services/resources running in the middleware server. Used Ant build scripts to build and deploy application on web logic server. Wrote unit tests and integration tests using JUNIT to thoroughly test the various modules of the application. Developed front end pages as per the client requirement using AngularJS.  Resolved the bugs in the application by reading the log files.  Configured and used Log4j for logging. Used SVN for version control of source code. Developed the application using Eclipse IDE. Provided support for system testing and user acceptance testing. Participated in code reviews and design discussions.

 Responsible for Functional Specification and System Design based on Business Requirement Document provided by business analyst. Actively participated in design and technical discussions.  Designed and developed client side GUI using JSP, HTML and JavaScript.  Used Java core concepts Collection Framework Interfaces like List, Set, Queue and also Map Interface.  Used JavaScript for validations and integrating business server side components on the client side with in the browser.  Used servlet as an interface between backend to frontend to handle http requests and to send response back to the front end.  Worked on JDBC to connect to MySQL for implementing CRUD operations.  Responsible for configuring Apache Tomcat Application Server to access the Database by setting up data source and MySQL Pool.  Developed the business objects using EJB for calling data access objects.  Used NetBeans IDE to develop the Application.  Used SVN for version control across common source code used by developers.  Building Software modules using Apache Ant.  Used Log4J to capture the log that includes runtime exceptions.  Performed unit testing and Integration testing.