• Developed highly optimized Spark applications to perform various data cleansing, validation, transformation and summarization activities according to the requirements.• Data pipeline consists Spark, Hive and Sqoop and custom build Input Adapters to ingest, transform and analyze user behavior (clickstream) data.• Developed Spark jobs and Hive Jobs to summarize and transform data.

• Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, Pig, Sqoop and Zookeeper.• Developed multiple POC’s using Spark, Scala and deployed on the Yarn Cluster, compared the performance of Spark, with Hive and SQL.• Use Amazon Elastic Cloud Compute (EC2) infrastructure for computational tasks and Simple Storage Service (S3) as storage mechanism. • Capable of using AWS utilities such as EMR, S3 and Cloud Watch to run and monitor Hadoop and Spark jobs on AWS.• Worked on SQL queries in dimensional data warehouses and relational data warehouses. Performed Data Analysis and Data Profiling using Complex SQL queries on various systems.

• Developed Oozie workflow for executing Sqoop and Hive actions and worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources.• Performance optimizations on Spark and Python. • Diagnose and resolve performance issues in Spark.• Responsible for developing Python wrapper scripts which will extract specific date range using Sqoop by passing custom properties required for the workflow.

• Involved in the complete SDLC software development life cycle of the application from requirement analysis to testing.• Involved in designing Database Connections using JDBC.• Developed the business components (in core Java) used for the calculation module (calculating various entitlement attributes).• Created complex SQL Queries, PL/SQL Stored procedures, Functions for back end. • Prepared the Functional, Design and Test case specifications.• Involved in writing Stored Procedures in Oracle to do some database side validations. • Performed unit testing, system testing and integration testing