•Develop data pipelines for transforming streaming data from Kafka Streams and loading them into AWS S3 buckets and Snowflake data warehouse•Designed and developed near real time spark streaming solution for processing CDC events from Oracle Golden Gate and Ingesting them onto Amazon S3 for long term storage and streaming onto Snowflake for Tableau reporting for business analytics. •Defining and coding Airflow Orchestration jobs for processing batch workloads to be launched in AWS EMR. •Built a PySpark framework to detect and process data that failed data quality checks during migration to AWS S3, resolving over 5400 data rejection errors for about 1500 datasets, reprocessing data and preventing data loss

•Responsible for ingesting large volumes of user behavioral data and customer profile data to Analytics Data store. •Developed custom multi-threaded Java based ingestion jobs as well as Sqoop jobs for ingesting from FTP servers and data warehouses.•Developed Scala based Spark applications for performing data cleansing, event enrichment, data aggregation, de-normalization and data preparation needed for machine learning and reporting teams to consume. •Worked on troubleshooting spark application to make them more error tolerant.•Worked on fine-tuning spark applications to improve the overall processing time for the pipelines.

•Developed Spark Applications by using Scala and Implemented Apache Spark data processing Project to handle data from various RDBMS and Streaming sources.•Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins and Transformations.•Used Spark for implementing the transformations on the historic data.•Experience with PySpark for using Spark libraries by using Python scripting for data analysis and aggregation and for utilizing data frames, developed Spark SQl API for processing data.•Used Spark programming API over EMR Cluster Hadoop YARN to perform various data processing requirements.

•Responsible for building scalable distributed data solution using Hadoop Cluster environment with Cloudera distribution.•Convert raw data with sequence data format, such as Avro and Parquet to reduce data processing time and increase data transferring efficiency through the network.•Worked on building end to end data pipelines on Hadoop Data Platforms.•Worked on Normalization and De-normalization techniques for optimum performance in relational and dimensional databases environments.•Designed developed and tested Extract Transform Load (ETL) applications with different types of sources.

•Installed and configured Apache Hadoop to test the maintenance of log files in Hadoop cluster.•Installed and configured Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster.•Installed Oozie workflow engine to run multiple Hive and Pig Jobs.•Setup and benchmarked Hadoop/HBase clusters for internal use.•Developed Java MapReduce programs for the analysis of sample log file stored in cluster.•Developed Simple to complex Map/reduce Jobs using Hive and Pig•Developed Map Reduce Programs for data analysis and data cleaning.

•Designed and developed applications using Spring MVC framework with Agile Methodology.•Developed JSP and HTML pages using CSS and JavaScript as part of the presentation layer.•Hibernate framework is used in persistence layer for mapping an object-oriented domain model to database.•Developed database schema and SQL queries for querying, inserting, and managing database.•Implemented various design patterns in the project such as Data Transfer Object, Data Access Object and Singleton.