I am a Senior Data Engineer with over 8+ years of experience designing and implementing streaming and batch data pipelines that power large-scale data solutions. My expertise spans data architecture, real-time processing frameworks, and data lakes, leveraging tools like Apache Spark, Flink, Beam, and Kubernetes.I specialize in designing and implementing cloud-based streaming frameworks to process millions of events in real time, and batch pipelines for large-scale data processing. My work includes designing and implementing scalable Data Lakes, optimizing query performance for faster insights, and scaling Kubernetes clusters using Prometheus for monitoring and Grafana for visualization. I have also designed and implemented distributed architectures by transitioning legacy models to modern solutions using Flink, Docker, and Kubernetes, ensuring scalability, reliability, and efficiency.I have designed and implemented metadata-driven frameworks for data migration and developed ingestion pipelines using Apache NiFi and Spark. My experience includes optimizing Spark job performance by caching and reusing intermediate data, as well as implementing data lineage tracking for seamless data flow understanding.Additionally, I have managed containerized applications with Amazon ECR, scaled infrastructure for Elasticsearch clusters, and created dynamic platforms for real-time deployment of Beam Flink clusters on Kubernetes. Leveraging tools like FluxCD, I ensure automated and reliable GitOps-based deployments for both batch and streaming solutions.Core Skills:Big Data & Cloud Tools: Apache Spark, Flink, Beam, Kafka, Hive, Hadoop, AWS, Kubernetes, EKS, Elasticsearch.Pipeline Expertise: Streaming and Batch Data Pipelines.Monitoring & Visualization: Prometheus, Grafana.Programming Languages: Java, Scala, Python, SQL.DevOps & Automation: Docker, Jenkins, FluxCD, Operator Patterns, NiFi.I am passionate about building innovative, scalable solutions for both streaming and batch data processing, unlocking the full potential of data for real-world impact. Let’s connect to collaborate on impactful projects and shape the future of data engineering together.
Listed skills include C++, Networking, Theory Of Computation, Algorithm Analysis, and 8 others.