- Design end-to-end custom technical solutions that meet the customers requirements in different industries- Engage with customers directly to conduct business and data discovery workshops, analyze dependencies and gaps, provide insights and conclusions based on the data- Design the data model by integrating datasets from different sources and tune structured and non-structured data - Develop custom data processing solutions through C3's IPO (for batch and stream data) using AWS Lambda, EC2, Kinesis, DynamoDB, specific to the customer- Manage storage, deployments, and internal connectors to cloud storage (S3, Azure, Hadoop)- Provide ETL and analytics solutions to customers using reporting tools such as Tableau

Analyze and define client’s strategy and determine system architecture and requirement to achieve goals.• Associated with creating Hive Tables, and loading and analyzing data using Hive Queries for reports.• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD's.• Work with a variety of data formats such as JSON, Compressed CSV, Parquet etc.• Worked in AWS environment for development and deployment of custom Hadoop applications.• Strong experience in working with ELASTIC MAPREDUCE(EMR)and setting up environments on Amazon AWS EC2 instances.• Very good experience with version control.• Importing and exporting data into HDFS and HIVE using Spark and Scala.• Load and transform large sets of structured, semi structured and unstructured data.• Developed job flows to automate the workflow for the data to be loaded to AWS S3 buckets and hive. • Designed and Implemented Partitioning (Multi-level), Buckets in HIVE.• Loaded the aggregated data onto Amazon S3 Buckets from Hadoop environment for reporting on the dashboard. • Performed ETL operations such as transformations, event joins, filters both traffic and some pre-aggregations before storing the data onto cloud.• Good understanding of Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance. • Responsible for the design and development of high performance, robust software using Scala• Developed different kinds of reports using large amounts of data that can be integrated with the app developed.• Expertise in writing the Scala/Python code using higher order functions for the iterative algorithms in spark for performance consideration.• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.• Worked in utilizing spark machine learning techniques implemented in Scala and Python.

Cleaned large dataset of nearly 11 million records and extracted smaller data sets using SQL Server• Created databases using SQL scripts and manipulated data using SQL queries utilizing aggregation and advance joins• Combined multiple data interfaces to derive final dataset with necessary projections mapped to requirements• Strategized and developed data-marts to assist the NGO streamline their metrics and help in decision making