• Research and recommend suitable technology stack for Hadoop migration considering current enterprise architecture. • Responsible for building scalable distributed data solutions using Hadoop. • Experienced in loading and transforming of large sets of structured, semi structured and unstructured data. • Developed Spark jobs and HiveJobs to summarize and transform data. • Involved in converting Hive/SQL queries into Spark transformations using Sparkdataframes, Scala and Python. • Expertise in implementing SparkScala application using higher order functions for both batch and interactive analysis requirement. • Experienced in developing Spark scripts for data analysis in both python and scala. • Wrote Scala scripts to make spark streaming work with Kafka as part of sparkKafka integration efforts. • Built on-premise data pipelines using kafka and spark for real time data analysis. • Created reports in TABLEAU for visualization of the data sets created and tested native Drill, Impala and Spark connectors. • Implemented Hive complex UDF's to execute business logic with Hive Queries. • Responsible for loading bulk amount of data in HBase using MapReduce by directly creating H-files and loading them. • Developed different kind of custom filters and handled pre-defined filters on HBase data using API. • Evaluated performance of SparkSQL vs IMPALA vs DRILL on offline data as a part of poc. • Worked on solr configuration and customizations based on requirements. • Implemented Spark using Scala and utilizing Data frames and SparkSQLAPI for faster processing of data. • Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS. • Exporting of result set from HIVE to MySQL using Sqoop export tool for further processing. • Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.

• Involve in all phases of SDLC (Software Development Life Cycle) which includes requirement collection, design and analysis, development and deployment of the application.• Architecture and design of business requirements and to make Visio Diagrams for the design and to develop the application and deploy the application in various environments. • Develop Spark 2.1/2.4 Scala component to process the business logic and store the computation results of 10 TB data into HBase database to access the downstream web apps using Big SQL db2 database.• Uploaded and processed more than 10 terabytes of data from various structured and unstructured sources into HDFS using Sqoop and Flume.• Test the developed modules in the application using Junit Library and Junit testing Framework• Analyze structured, unstructured data, and file system data and loading the data to HBase tables based on the project requirement using IBM Big SQL with Sqoop mechanism and processing the data using Spark SQL in-memory computation &processing results to Hive, HBase• Handle importing other enterprise data from different data sources into HDFS using JDBC and Load Hadoop in Big SQL and perform transformations using Spark API to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from upstream in near real time and persists into HBase.• Working with different file structures with different Hive file formats like Text file, Sequence file, ORC file, Parquet and Avro to analyze the data to build data model and reading them from HDFS and processing through parquet files and loading into HBASE tables.

• Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into MongoDB. • Developed Spark scripts by using Scala shell commands as per the requirement. • Used Spark API over Hortenworks to perform analytics on data in Hive. • Developed Scala scripts using both Data frames/SQL and RDD/MapReduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop. • Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning. • Loaded the data into Spark RDD and do in memory data Computation to generate the Output response. • Optimizing of existing algorithms in Hadoop using SparkContext, Spark-SQL, DataFrames and PairRDD's. • Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.

• Worked on analyzing Hadoop cluster and different Big Dataanalytic tools including Pig, Hive HBase database and SQOOP. Involved in Unit testing and delivered Unit test plans and results documents. • Collected and aggregated large amount of web log data from various sources such as webservers, mobile and network devices using ApacheFlume and stored the data into HDFS for analysis. • Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose. • Installed Hadoop, Map Reduce, HDFS, and Developed multiple map reduce jobs in PIG and Hive for data cleaning and pre-processing. • Installed, monitored and maintained hardware/software related issues on Linux/Unixsystems. • Investigated, installed and configured software fail-over system for production Linux servers. • Designed, developed, debug, tested and promoted Java/ETL code into various environments from DEV through to PROD.

• Installed and configured SQL server 2005 and set-up Various Databases.• Installed and configured Oracle databases on Unix and Linux platform.• Created all administration tasks of SQL Server and designed database• Worked with development team in relation to application tuning.• Created tables, views, and indexes.• Developed various stored procedures, cursors, functions, and triggers. • Created tables and provided constraints and user defined rules and defaults.• Used system stored procedures and DBCC commands.• Created Database Maintenance Planner for the Performance of SQL Server which covers Database Integrity Checks, Update Database Statistics and Re-indexing. • Scheduled backups and performed restore operations for system test data depends on testers requirement.