Designed and implemented highly efficient, scalable data pipelines within Hadoop ecosystem, employing HDFS, MapReduce, Hive, and Sqoop that resulted in a notable 30% enhancement in pipeline efficiency.Designed and implemented Incremental DBT models for UPI, IMPS, BHIM, and International UPI transactions, resulting in a substantial 20% increase in data accuracy and operational efficiency.Migrated 50 TB data from Hadoop to MinIO (S3), ensuring seamless compatibility and boosting data accessibility by 40%.Implemented Directed Acyclic Graphs (DAGs) with Dagster to refine ETL workflows, resulting in a notable 70% decrease in manual intervention and an outstanding 70% enhancement in data processing speed.

Designed and optimized SQL queries to analyze system performance metrics for UPI, BHIM, and IMPS transactions, leading to a 20% reduction in query response time.Implemented automated data processing workflows with Apache Airflow, achieving a 30% decrease in manual intervention and ensuring reliable execution of critical data pipelines for analytical insights.Enhanced Hive SQL query performance by 20% through collaborative efforts with cross-functional teams using Hue's interface, providing comprehensive data insights crucial for informed decision-making.Implemented complex SQL queries and fortified the data framework for anti-money laundering (AML) in conjunction with UDIR systems to support Unified Payments Interface (UPI) functionalities. These efforts directly resulted in a commendable 20% reduction in fraudulent transaction incidents, showcasing proficiency in database optimization and fraud detection.