Currently building team to work on Enterprise level standardization of Kafkausage/use-caseLead a team to build the cost utilization of snowflake jobs.Designed and lead a POC for migrating data from rdbms to snowflake usingKafka and Kafka connectReinvent the existing jobs to mongo dB, which helps in reduction of 30% oftime.Worked as a part of architecture team to establish NiFi cluster on the EKSsystemRe-structed the existing on-prem NiFi cluster automation to AWS ec2 clusterautomation using Jenkins jobs.Leading role in leading designing POC on multiple ELT to

Spark job and pipeline optimization, which lead to 3hrs of save in jobexecution.Lead a part of team for leading bank to build bigdata pipeline havingCustomer 720 degree information.Member of the architecture team which analysed and designed the complexdata information accross multiple channel.Lead a small team to migrate bank data from informatica system to Hadoophdfs.Redisgned the data visualization in hdfs using hive concept.

Developed generic data extraction framework in sqoop and shell script. The framework was deployed in many other projects.Built the audit framework in hive which can be integrated in any layer (extraction and transformation)Re-build the SAS transformation in Pig Latin language and build the datalake in HDFS.Hive table with required business logic was built for intermediate data visualisation.Built POC application on data stream which the help of kafka and spark streaming.

Worked extensively on data extraction with the help of Sqoop and building reconciliation login in shell scriptDeveloped the data transformation in complex hive query and build the dimension and fact layer.Extensively automated many application process in shell script Build POC application in spark rdd, which latter helped in getting many projects component.

Responsible for writing and debugging complex sql in oracle.Developing shell script for data validation and job scheduling.Debugging of medium size java application.