- Key team member of LinkedIn's stream processing platform which handles 8000+ pipelines processing trillions of events per day using Apache Samza and Apache Beam.- Led a massive cross-org migration effort of all streaming jobs from Kafka to Linkedin’s nextgen pub-sub system as a part of company-wide infra modernization initiative.- Leading the effort to migrate job state-backup support from Kafka to LinkedIn’s S3-like blob-store which has proven to reduce job down-time across restarts by over 10x.- Authored multiple build, tooling and user-facing control plane improvements. Some examples include- gradle plugins & repo templates for creating jobs, packaging and config publishing, job reset tool for easy job reset and drain tool for seamless schema evolution- Created pipeline authoring APIs and language-agnostic protobuf DSLs from core features like source & sinks to complex features like batch lookups & partial writes to KV stores.- Mentored engineers and interns to explore important initiatives like batch-stream convergence,Beam’s tensor flow evaluation for language-agnostic pipeline authoring for LinkedIn’s ML use cases- I am also a committer in the open source Apache Samza project:https://samza.apache.org/community/committers.html

• Built the streaming data platform at Godaddy using Apache Beam and Flink - Created data sources and sinks for Hive, JDBC, json data - Key sdks and transforms - Data marshalling using protobufs• Creating the deployment infrastructure and framework using AWS. The framework- - Provisions AWS services using Cloudformation templates, sceptre and boto3. - Uses helm templates to deploy dockerized applications on AWS. - The framework also enables developers to test their cloud applications on their local machines using Localstack and Docker Desktop's local k8s cluster

Spark data pipelines for machine learning models for credit card fraud detection

• Modified data pipelines by tuning existing Hive and Pig code base for crucial data marts. Migrated critical code to Spark (PySpark) leading to over 60-70% performance improvement• Created multiple workflows and schedules using Oozie incorporating Hive/Pig/Spark actions and replacing existing Cron jobs