Implemented Apache Kafka to handle real-time data streams, enhancing data ingestion efficiency and scalability. Reduced data latency by 70% through effective message queuing and processing.Engineered ETL pipelines using Python and Apache Airflow, ensuring seamless data extraction, transformation, and loading processes. Enhanced data quality and consistency while optimizing workflow automation.Managed SQL (MySQL) and NoSQL (Cassandra) databases, optimizing them for performance and scalability. Implemented indexing strategies and schema optimizations for efficient data storage and retrieval.Implemented machine learning models, such as regression and classification algorithms, integrated with Apache Flink for real-time data analysis, enhancing predictive capabilities and adaptability over time.Utilized Apache Flink for real-time data streaming, processing approximately 500,000 records per hour. Significantly reduced data latency, enabling timely insights and decision-making for stakeholders.Developed interactive dashboards and reports using Power BI and MS Excel, providing stakeholders with intuitive insights into key metrics and trends. Enabled data-driven decision-making and cross-functional collaboration.

India Orchestrated ETL operations using Azure Data Factory and Azure Databricks, processing over 10TB of data monthly from various sources to Azure storage services including Azure Data Lake, Azure SQL, and Azure Data Warehouse.Successfully implemented and managed Azure Data Lake platforms, handling the extraction and integration of data from 15+ upstream systems, resulting in a 20% increase in data retrieval efficiency.Engineered and maintained 30+ pipelines in Azure Data Factory, enhancing the ETL process for diverse data sources like Azure SQL and Blob Storage, leading to a 25% reduction in processing time.Developed and optimized 20+ Spark applications using Scala and Spark-SQL, facilitating a 30% improvement in data processing speed and efficiency for complex data sets.Expertise in integrating data from NoSQL (MongoDB) and SQL databases (MySQL), using Debezium for real-time data streaming into Kafka, handling over 5TB of streaming data per month.Employed Airflow for batch processing, managing data transfers of approximately 8TB per month into Azure Data Lake Storage, streamlining data management and analytical processes.Implemented advanced Partitioning and Bucketing techniques in HIVE, leading to a 10% improvement in data query performance and organization.Enhanced data availability and usability by 30% through Hive query refinement, alongside implementing diverse HDFS file formats and compression techniques for a 15% increase in storage efficiency.