• Implemented Apache Kafka for real-time data streaming, improving data ingestion speed and reliability.• Managed, configured, and scheduled resources across the cluster using Azure Kubernetes Service (AKS), enabling seamless orchestration of containerized data processing applications for enhanced scalability.• Containerized data processing applications using Docker to ensure consistency across environments, streamlining the deployment of data ingestion pipelines on Azure HDInsight Spark clusters and facilitating real-time data transformation with Spark SQL.• Developed and maintained RESTful APIs to facilitate seamless communication between microservices.• Designed and implemented microservices architecture to enhance scalability and performance of applications.• Implemented microservices architecture on Kubernetes, optimizing high availability and fault tolerance for data ingestion and transformation processes, including large data sets in Cosmos DB (SQL API and Mongo API).• Developed data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL, while ensuring efficient resource allocation via automated deployment and scaling of Spark clusters using Kubernetes.• Developed dashboards to help business users analyse data as well as provided data insights to upper management, utilizing Microsoft products like SQL Server Reporting Services (SSRS) and Power BI.• Performed the migration of large data sets to Databricks (Spark), administering clusters, loading data, and configuring data pipelines from ADLS Gen2 to Databricks using ADF pipelines, ensuring real-time data ingestion with Spark Streaming.• Monitored and optimized containerized applications on Kubernetes clusters with tools like Prometheus and Grafana, ensuring optimal performance of Verizon’s critical data workflows.