M Singh

M Singh Email and Phone Number

Principal Engineer (Stream processing) @ Cybersecurity Company
Philadelphia, PA, US
M Singh's Location
Philadelphia, Pennsylvania, United States, United States
About M Singh

Certifications: - Confluent Certified Developer for Apache Kafka - Databricks Certified Associate Developer for Apache Spark 3.0 Open Source Contributor: Apache Flink - Retention policy for InfluxDB metrics - Error reporting for Datadog metrics Apache Airflow - Redis/AWS SQS Publish OpsApache NiFi - AWS Kinesis, Firehose, Lambda, DynamoDB, Http proxy, Creds prov - RethinkDB, InfluxDB, ElasticSearch, Neo4J, Ignite - DL4J (pr) - Expr lang Apache Big Data Projects - Apache Seatunnel - Apache DolphinScheduler Other open source projects (bug fixes/docs) - vector - Localstack-java-utils - ververica/Flink Sql Cookbook - h2databaseWasabi - A/B Testing Srv - Implemented data access tier using Datastax driverProfessional Summary26+ years exp in software dev with open source techs/patterns/fwks12+ years exp with Big-Data techsExpertise in developing near real time (nRT)/event driven/analytic apps (Flink/Spark/Kafka/ES/RethinkDB/InfluxDB/Neo4J)Experience with SQL Eng (Trino/Drill)Exp with Kafka eco-sys - Kafka/KStreams/KSQL/Schema Registry/Rest ServiceExpertise in Flink eco-sys (Apache Flink/Flink Stateful Functions)Hands on exp with Airflow hooks/sensors/processors/dynamic EMR DAGsExpertise in extending Apache Spark Structured/Streaming/Flink sources/sinksExp in implementing streaming A/B testingHands on experience w/ AWS for batch/RT processing (S3/DynamoDB/Kinesis/Lambda/EMR/EC2)Exp in using testcontainers for container testingExp in integrating Datadog, Prometheus, Grafana & Consul for monitoring streaming appsExpertise in integrating ETLs w/ NoSQLs (Cassandra/ElasticSearch/DynamoDB/RethinkDB/InfluxDB/Neo4J)Exp w/ in-mem grid (Apache Ignite), TS DB (InfluxDB), Graph DB (Neo4J)Expertise: Big-Data Techs, distributed processing, ETL (Stream/Batch)Real Time/Stream processing and extending Apache Spark/Apache Flink/Apache Flink Stateful FuncConfluent Kafka eco system - Kafka/KStreams/KSQL/Schema Registry/Rest ServiceETL orches - Airflow hooks/sensors/processors/DAGsData flow frameworks (Apache Flink/Apache Spark/Apache NiFi)SQL Engines - Trino/Apache DrillIn-Memory Data/msg/compute grids (Apache Ignite)NoSQL (Cassandra, HBase, DynamoDB, ElasticSearch, RethinkDB, Redis)Cloud Computing (AWS S3, Kinesis, Lambda, DynamoDB, EMR/Spark/Flink)Distributed message broker (Apache Kafka, AWS Kinesis)Streaming A/B testingMetrics systems - Prometheus, DatadogContainers - Dockers/KubernetesTime-Series DBs -InfluxDB, PrometheusPatterns - DP, EIP

M Singh's Current Company Details
Cybersecurity Company

Cybersecurity Company

View
Principal Engineer (Stream processing)
Philadelphia, PA, US
M Singh Work Experience Details
  • Cybersecurity Company
    Principal Engineer (Stream Processing)
    Cybersecurity Company
    Philadelphia, Pa, Us
    View
  • Cybersecurity Company
    Principal Engineer
    Cybersecurity Company Dec 2021 - Present
    Implemented Kubernetes Deployments/Jobs/Services for Datadog/Vector data pipeline with http scrape/VRL/elasticsearch componentsImplemented Kafka to ServiceNow table integration using Datadog Vector based streaming pipelineImplemented AWS Lambda based event driven pipelines
  • Realogy
    Sr Data Engineer (Stream Processing)
    Realogy Jun 2021 - Dec 2021
    Implemented streaming etl pipeline using Apache Flink/Kafka on AWS EMR & AWS Kinesis Data Analytics Platform (Delivered ETL code with 99+% code/line coverage)Integrated ETL pipeline with Kafka schema registry to ingest Avro messagesImplemented streaming entity resolution with remote mdm using async apisImplemented Apache Flink ETL pipeline to save data to Snowflake lake-houseRe architected Kafka based real-time etl ingestion pipeline to reduce complexityLead the effort to standardize Avro message with metadata and uniqueness identifierContributed to open source projects including docs for Apache Flink
  • Ampersand
    Sr Data Engineer (Spark Processing)
    Ampersand Dec 2020 - Jun 2021
    Implemented multi-staged ETL pipelines for building enterprise data lake using Apache Spark/EMRApplied intentional programming techniques to make the ETL pipeline stages testableImplemented approximate re-aggregable cardinalities using sketch librariesStreamlined storage integration to use declarative Apache Spark data sink apisIdentified data quality issues
  • Consulting Company (Turnberry Solutions)
    Sr Big Data Engineer (Client - Telecom/Media)
    Consulting Company (Turnberry Solutions) May 2019 - Dec 2020
    Extended Apache Flink Datadog metrics framework to capture application specific metricsImplemented gauge and filtering capabilities for Apache Flink & integrated with Datadog reporterDeveloped components for real-time event driven application (Apache Flink/AWS EMR)Lead the effort and resolved issues to migrate to Flink 1.10 on AWS EMRStandardized and automated Datadog dashboard creation for metrics collectionDesigned and lead the effort to enable distributed event tracing for Apache Flink App & ElasticSearchContributed enhancements to Apache Flink's Datadog and InfluxDB metrics frameworks, and localstack-java-utils sample
  • Thoughtworks, Inc
    Lead Data Engineer (Clients - Telecom/Finance)
    Thoughtworks, Inc Jan 2019 - Apr 2019
    Implemented Apache Airflow processor for event driven DAGsArchitected migration from in-house DWH ETL to AWS EMR/Airflow based data lake ETL platformImplemented components for Apache Spark ETL pipelineMember of business development and architecture review teams for Data Engg engagements
  • Pinpoint Predictive
    Sr Data Engineer
    Pinpoint Predictive Oct 2018 - Jan 2019
    San Mateo, California, Us
    Implemented Apache Airflow DAG for ETL pipelinesCreated custom Airflow continuous & stateful sensors for event driven ETL pipelinesImplemented ad-hoc Spark analytics applications
    View
  • Samsung Electronics America
    Staff Software Engg Iii
    Samsung Electronics America Mar 2017 - Oct 2018
    Ridgefield Park, Nj, Us
    Implemented real-time ingestion & customized sessionization pipeline using Apache Spark Structured Streaming, Kafka and streaming JDBC sinkImplemented Airflow workflow DAGs Designed and implemented metrics dashboards using Apache Spark/Elasticsearch & KibanaImplemented Apache Spark & AWS S3 based batch processing pipeline with custom sessionization/deduping/enrichment for analytics applications using bronze/silver/gold ETL modelImplemented Apache Spark/Cassandra-connector application to migrate data from MySQL to CassandraContributed RethinkDB/InfluxDB/Elasticsearch processors to Apache Nifi
    View
  • Intuit
    Staff Software Engineer
    Intuit Jan 2015 - Mar 2017
    Contributed AWS Kinesis, AWS Lambda, AWS Kinesis Firehose, AWS DynamoDB processors to Apache NifiIntegrated/contributed Apache Ignite cache processor with Apache Nifi Extended Apache Nifi data flow processors (AWS S3/SQS/SNS processors) for using creds providerDeveloped data flows using Apache Nifi (Http REST/S3/RDBMS to Hive as Scoop replacement)Implemented real-time ETL pipeline & visualization using Kafka, Spark & ElasticSearchDeveloped RTAP app with ElasticSearch/Logstash/Kibana (ELK) for app metricsDesigned and implemented RestFul ingestion pipeline using Spring and Java StreamsLead the effort to use ELK stack as a unified platform for log/metric analytic/application developmentDeveloped Datastax native Cassandra driver accessors for supporting product requirementsIntroduced engineers to Apache Spark/Apache FlinkDeveloped Spring-data-cassandra and Spring-Boot based POC for RESTFul services as an alternative to in-house frameworkReceived team/individual Spot-Light Awards (Win-Together, Learn-Fast, Deliver-Awesome)
  • Consulting Company (Triple Crown Consulting)
    Big Data Engineer/Bi Architect (Client - Telecom/Media)
    Consulting Company (Triple Crown Consulting) Feb 2014 - Dec 2014
    Designed/developed streaming data ETL pipelines (Spark/Cassandra/Kafka)Implemented custom Spark streaming receiverImplemented real-time and batch integration between Cassandra & ElasticSearchDeveloped RTAP POC with ElasticSearch/Logstash/Kibana (ELK)Implemented event aggregation analytics app with Cassandra/Cql/pigDeveloped pig scripts for aggregate data validation
  • Impetus Tech
    Data Architect (Client - Financial)
    Impetus Tech Jul 2013 - Feb 2014
    Designed Hbase schema & implemented Hbase/Spring-hadoop library for optimal R/WImplemented strategy for HBase based multi-tenancy schema for different envsDeveloped batch/integration workflow using Spring Batch/Hbase/REST svcDeveloped Pig/HBase report scriptsDeveloped nRT distributed analytics application using Storm/JMS/HBase
  • Consulting Company (Insight Global)
    Sr Software Engineer (Client - Financial)
    Consulting Company (Insight Global) Oct 2012 - Jun 2013
    Developed Pig scripts & UDFs for user interaction/experience analysisImplemented integrated Hive & Pig based reporting scripts for A/B testingImplemented Hive UDxs for extracting data from Hadoop for internal appsDesigned and implemented system integration applications using Mule ESB
  • Comcast
    Tech Lead/Principal Engineer
    Comcast Apr 2007 - Sep 2012
    Philadelphia, Pa, Us
    Lead a team to develop real-time/streaming event processing application using Twitter Storm/CassandraDesigned/implemented cloud (AWS-EC2) based cross data-center elastic data processing middleware using Zookeeper (POC) - later used for in house multi-data center highly available/fault tolerant Storm clusterDeveloped REST/ROA based scalable middle-ware for appsLead the team/management to adopt Splunk for analyzing logsDeveloped middle-ware integration adapters for 3rd party services Implemented data-driven high performance search app using SolrDeveloped mock strategies to support early integration testingMentored interns on design patterns/testing/JEE app development
    View
  • Traffic.Com
    Software Architect/Engineer
    Traffic.Com Mar 2006 - Apr 2007
    Developed distributed cache co-ordination framework using JGroups/JMXImplemented JMX monitoring for standalone/managed apps with Hyperic Developed fault tolerant persistence for JMS serviceDeveloped JMS service facade for client applicationsDeployed cruise control for continuous integration & builds
  • Siemens Medical
    Software Engineer, Senior Staff
    Siemens Medical Apr 2004 - Mar 2006
    Lead team/management to use detached persistent domain objects (JDO)Developed Rich Client Platform components for Eclipse based application Designed/lead a team to develop static byte-code analysis toolInstrumented Java classes for performance monitoring
  • Recare
    Senior Software Engineer (Client - Recare)
    Recare Jun 2003 - Apr 2004
    Designed/developed GoF adapter pattern based web service framework for integrating practice management apps Enhanced Java Swing based content editor
  • Health Language Inc, Co
    Content Software Architect/Engg
    Health Language Inc, Co Jul 2000 - Jun 2003
    Designed/developed ETL process to manage/update clinical terminologiesDeveloped content validation tools
  • Cerner Corp
    Application Developer
    Cerner Corp Aug 1998 - Jul 2000
    Kansas City, Missouri, Us
    Created image driven selection/editing tool based on ATL/COMDeveloped front end for web based search tool
    View
  • Ontyx, Inc
    Member Of Technical Staff /App Developer
    Ontyx, Inc Aug 1997 - Aug 1998
    Developed client-server applications (JDBC/RMI) to access EISCreated a knowledge base for pharmacological terms (anti-hypertension drugs) using Ontylog tools
  • Chref, Inc
    Data Analyst
    Chref, Inc Jan 1996 - Aug 1997
    Developed clinical outcome reports for various medical conditions/procedures using ICD9Developed trauma patient outcome report based on injury severity score (ISS)

M Singh Education Details

  • New Jersey Institute Of Technology
    New Jersey Institute Of Technology
    Computer Science
  • New Jersey Institute Of Technology
    New Jersey Institute Of Technology
    Electrical Engineering

Frequently Asked Questions about M Singh

What company does M Singh work for?

M Singh works for Cybersecurity Company

What is M Singh's role at the current company?

M Singh's current role is Principal Engineer (Stream processing).

What schools did M Singh attend?

M Singh attended New Jersey Institute Of Technology, New Jersey Institute Of Technology.

Free Chrome Extension

Find emails, phones & company data instantly

Find verified emails from LinkedIn profiles
Get direct phone numbers & mobile contacts
Access company data & employee information
Works directly on LinkedIn - no copy/paste needed
Get Chrome Extension - Free

Aero Online

Your AI prospecting assistant

Download 750 million emails and 100 million phone numbers

Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.