Working as Data Engineer for client (Citi Bank) to store, process and manage huge amount of data in day to day and real time operations collected from various heterogeneous sources.Project: Nov 2018 - Present• Responsible for developing end to end program.• Created a Spark batch job to read XML file from a file path, enriching and extracting critical data.• Involved in moving file details from HDFS to Kafka• Developed Spark streaming job which processes real time messages from Kafka which are in JSON format.• Created the Spark job to match and enrich data from different sources and sending data through Kafka• Involved in creating and maintaining HBase Table to store data from different Sources.• Developed Shell Scripts to run various jobsProject: Aug 2019 – Present• Created spark script to flatten complex view to store it in Impala integrating shell scripts & Hive.• Worked on performance optimisation of Impala view and Spark Job, reducing job time by 200%.• Developed HQL Script for analysis of data in Hive Tables• Performed data enrichment such as filtering, format modelling, sorting and aggregation• Developed managed and external Hive tables with appropriate partitioning

Trained in Python, SQL, Microsoft Server (SSIS, SSRS, SSAS), Big Data Hadoop, Hive, Pig, Scoop, MongoDB, SparkDesigned Multi Dimensional Data Model, Created an ETL process to load the data in a cube, Created interactive dashboard in Microsoft SSRS that generates reports based on the role and criteria selected.

Did 2 projects there1.Developed a webpage for processing of raw data generated from various network softwares using Perl.2.Designed a Robust Transmission backhaul network for a cell site of greenfield town