• Developed Spring Boot application for efficient ScyllaDB query executions by implementing data access layer and exposing REST APIs, used Jackson databind for JSON handling and dataStax OSS driver for data access.• Integrated Spring Security for role-based access control (RBAC) in Kafdrop, a Kafka monitoring tool ensuring authorized access and compliance with organizational policies for kafka clusters data.• Implemented Caffeine caching in the Springboot application to optimize performance and reduce latency, configured with expiration policies and size constraints for efficient handling of Kafka metadata• Optimized data transmission from JSON to Avro format in Kafka, reducing payload size and improving throughput by 30%, cutting data storage and bandwidth costs significantly• Led adoption of Avro serialization and Schema Registry of Confluent ecosystem with dynamic Avro schema generation from Java model classes , enhancing network and disk I/O efficiency by 40%.• Streamlined data pipeline by Implementing MySQL Debezium,ScyllaDB and ElasticSearch connectors with Kafka Connect by transitioning from custom-built microservices and Flink jobs and optimizing data processing with custom SMT(transformers).