Big Data Batch and Streaming @ Visa AI Platform using Kafka, Flink, Hadoop, Spark, Hive, Presto, Spring Boot, Apache Camel, S3, Java, Scala, shell scripting, Python, SQL and etc.1. Data streaming team in the Visa AI Platform, using Kafka and Flink for real time low latency pipeline and Spark and Hadoop for the batch pipeline. Job responsibility include software development, testing and release, architectural design, performance tuning, junior member mentoring, and prod on call support.2. Lead the real time Flink data streaming pipeline development and maintenance, which includes:• Data ingestion and insertion from/to various data sources including kafka, s3 and hdfs, enabled with kerberos authentication through the Flink JaaS module.• Using Flink DataStream API for data cleaning and feature generation, data stream joining, windowing and deduping, asynchronous http call for model state updates and etc.• Performance and system optimization including aligned and unaligned checkpointing analysis, backpressure reduction and optimization, heap analysis and JVM garbage collector optimization, network buffer tuning, rocksDb performance optimization and etc.• Monitoring with Flink built in metrics and UI, as well as integrating with Prometheus and Grafana.3. Contributor of the batch data pipeline using Spark sql API for development and job scheduling tool4. Lead and the sole contributor to one Java based data pipeline using Springboot, Apache Camel and s3, responsible for end to end delivery including development, unit test, integration test, and release.

Full stack development @ Payment System Risk using Angular, Spring, MySQL and etc.1. Full stack web development of a near real-time risk monitoring and alerting application using Angular, Spring and MySQL.2. Lead of the project, managing end to end development cycle including feature discussion and planning, code development and testing, release support, security finding remediation, server and environment setup, etc. Responsible for junior member mentoring.3. Major features include events publishing and consuming from Kafka, events processing and evaluation engine, data ETL using hive, shell scripting and Dseries scheduler, auto-populating alert summary and notification email, searching capabilities, spreadsheet events processing from mailbox attachments, SSO, etc.