• Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.• Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.• Managed and reviewed Hadoop log files to identify issues when job fails and used HUE for UI based pig script execution, Oozie scheduling.• Involved in creating data-lake by extracting customer's data from various data sources to HDFS which include data from Excel, databases, and log data from servers.• Developed Python code to gather the data from HBase and designs the solution to implement using PySpark.• Developed PySpark code to mimic the transformations performed in the on-premise environment and analyzed the SQL scripts and designed solutions to implement using PySpark.• Automated workflows using shell scripts pull data from various databases into Hadoop and developed scripts to automate the process and generate reports.• Created detailed AWS Security groups which behaved as virtual firewalls that controlled the traffic allowed reaching one or more AWS EC2 instances.• Designed multiple Python packages that were used within a large ETL process used to load 2TB of data from an existing Oracle database into a new PostgreSQL cluster.• Deploy and configured cloud AWS EC2 for client websites moving from self-hosted services for scalability purposes and work with multiple teams to provision AWS infrastructure for development and production environments.• Designed number of partitions and replication factor for Kafka topics based on business requirements and worked on migrating MapReduce programs into Spark transformations using Spark and Scala, initially done using python (PySpark).System (HDFS) on Amazon EMR cluster by setting up the Spark Core for analysis work.

• Analyze and define researcher's strategy and determine system architecture and requirement to achieve goals and developed multiple Kafka Producers and Consumers from as per the software requirement specifications.• Used Kafka for log accumulation like gathering physical log documents off servers and places them in a focal spot like HDFS for handling.• Configured Spark Streaming to get ongoing information from the Kafka and store the stream information to HDFS.• Implemented Spark using Python and Spark SQL for faster processing of data and Worked on migrating MapReduce programs into Spark transformations using Spark and Scala, initially done using python (PySpark)• Implemented usage of Amazon EMR for processing Big Data across Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).• Involved in development of Hadoop System and improving multi-node Hadoop Cluster performance and responsible for developing data pipeline with Amazon AWS to extract the data from weblogs and store in MongoDB.• Developed Real time data processing applications by using Scala and Python and implemented Apache Spark Streaming from various streaming sources like Kafka, Flume and JMS.• Storing and loading the data from HDFS to Amazon S3 and backing up the Namespace data into NFS and integrated Hive server 2 with Tableau using Horton Works Hive ODBC driver, for auto generation of Hive queries for non-technical business user.

• Developed Simple and complex Map Reduce streaming jobs using Java language that have implemented them Using Pig.• Ingested data into HDFS from Oracle and vice-versa using Sqoop. • Extensively used Pig for Data cleansing.• Analyzed the data by running Pig Latin scripts to study customer behavior.• Handled structured and unstructured data and applying ETL processes.• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.• Developed the Pig UDF'S to pre-process the data for analysis.• Written multiple UDF programs in Java for data extraction, transformation and aggregation from multiple file formats (XML, JSON, and CSV).• Develop and maintained complex outbound notification applications that run on custom architectures, using languages Core Java, J2EE, SOAP, XML, JMS, JBoss and Web Services. • Experienced in monitoring and debug performance issues on Linux (RHEL & Centos).• Involved in Production Rollout Support which includes monitoring the solution post go-live and resolving any issues that are discovered prior to Rollout. • Integrating Hadoop with Kafka. Expertise in uploading Click stream data from Kafka to HDFS.• Designed and documented operational issues by following standards and procedures in a software reporting tool JIRA

• Worked on Hadoop Cluster with size of 83 Nodes and 896 terabytes capacity.• Worked on Map reduce jobs, HIVE, Pig.• Involve in Requirement Analysis, Design, and Development.• Importing and exporting data into Hive and HBase using Sqoop from existing SQL server.• Experience working on processing unstructured data using Pig and Hive.• Developed Pig Latin scripts to extract data from the web server output files to load into HDFS.• Implemented Partitioning, Dynamic Partitions, Buckets in Hive.• Developed Hive queries, Pig scripts, and Spark SQL queries to analyze large datasets.• Exported the result set from Hive to MySQL using Sqoop.• Created and maintained technical documentation for launching Hadoop clusters and for executing Hive queries and Pig Scripts.• Worked on debugging, performance tuning of Hive, Pig Jobs.• Gained experience in managing and reviewing Hadoop log files.• Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs.• Used NoSQL database with HBase.• Actively involved in code review and bug fixing for improving the performance.

• Involved in the complete Software Development Lifecycle (SDLC) using the Agile iterative development• Methodology. Interacted with the end users and participated in the SCRUM meetings.• Developing End-User friendly GUI using JSP, HTML, DHTML, JavaScript & CSS• Implemented CSS Manipulation, HTML Event functions, Java script Effects and animations using JQUERY.• Involved in development of application using struts, RAD and Oracle database.• Developed Data Access Layer using Hibernate ORM framework.• Have coded numerous DAO's using Hibernate Dao Support. Used Criteria, HQL and SQL as the query languages in Hibernate Mapping.• Used Web Services for transmission of large blocks of XML data using SOAP.• Used XML for data exchange and schemas (XSDs) for XML validation. Used XSLT for transformation of XML.• Written numerous test cases for unit testing of the code using JUnit testing framework.• Configured and Deployed application on Web Sphere Application Server.