Involved in gathering the requirements, designing, development and testing.Active Coordination with other teams like analytics, data scientists and product teams.Building data pipelines, data sets and conducting performance tuning using Azure databricks, Azure data lake gen2, Azure event hub services on Azure PlatformUsing response model in RestAPI to get data from AWS buckets and process data into Storage platforms using business requirement logics.Following Scrum Agile methodology (Sprint Planning, Daily Scrum Meeting, Backlog Refinement, Sprint Demo, Sprint Retrospective)Writing test cases using scalatest frameworks like FunSuite/FlatspecCreating jobs in Azure Databricks and Scheduling them using job scheduling tool AppworxUsing version control tool GIT

Involved in Design and Requirement Gathering Phase for the project.Working closely with the stakeholders & solution architect.Ensuring architecture meets the business requirements.Building highly scalable, robust & fault-tolerant systems.Improving data quality, reliability & efficiency of the individual components & the complete system.Creating a complete solution by integrating a variety of programming languages & tools together.Used Spark SQL Dataframe API and Aggregate functions extensively for transforming the raw data and created user defined functions (UDF’s)Used Scala Pattern matching, Higher Order Functions for writing generic re-usable codes.Code review and unit test case review provided technical assistance to the team.Scheduled spark action through Rundeck Scheduler.

Involved in gathering the requirements, designing, development and testingUsed Sqoop to ingest data from source into the Hadoop cluster.Created framework to perform ETL operations, using Scala IDEWriting shell scripts, which are responsible for triggering the Spark jobs.Workflow configuration of jobs using Autosys in SIT, UAT & ProdUnit testing and documenting, tracking, & communicating test plans, test results, analysis

 Analyzed large data sets by running Hive queries and Pig scripts.  Created Hive tables, and loading and analyzing data using hive queries. Involved in running Hadoop jobs for processing millions of records of text data. Involved in loading data from LINUX file system to HDFS. Responsible for managing data from multiple sources. Responsible to manage data coming from different sources. Exporting analyzed data to relational databases using Sqoop. Scheduling the jobs by using OOZIE.