Tech Lead of LinkedIn Anomaly Detection Platform• Spearheaded the next-generation Anomaly Detection platform, leading a team of 4 engineers to successfully implement the POC within 1.5 quarters, demonstrating effective management and strategic foresight.• Delivered key components on schedule, including new APIs, anomaly detection job, and event-based scheduling. Crafted and executed a detailed plan that dynamically adjusted to project needs, ensuring timely completion of development milestones.• Authored technical designs for architecture and core components, and contributed directly to critical development areas including third-party integration and the V2 Anomaly Detector.Tech Lead of LinkedIn Data Quality Platform• Spearheaded the Data Quality team’s strategy and roadmap, leading the creation of True North Metrics, User Adoption Strategy and Enforcement Strategy. Played a key role in OKR planning and proactively discussing progress and challenges with leadership, leading grooming sessions and communicating objectives clearly to DQ developers, ensuring clarity and cohesion in team efforts.• Played a pivotal role in expanding Data Quality Contract coverage and alerting activation, managing the activation of 2800+ datasets and orchestrating the delivery of 4000+ daily alerting emails. Actively collaborated with key teams and integrated user feedback to refine data quality products and strategies.• Addressed complex system issues to improve notification reliability from 3K+ failures daily to nearly zero, enhancing system stability and user trust. Led the design and execution of tests to validate the scalability of new features, ensuring robustness and efficiency in data handling.

Tech Lead of LinkedIn Audience PlatformNext-gen Audience Pipelines• Led the redesign of Marketing Org’s Next-generation Audience Pipelines, managing a team of 4 engineers over 4 quarters to deliver new pipelines driving $800M in revenue.• Directed the build and optimization of 5 data pipelines, reducing runtime by 50% and codebase from 73K to 17K lines, while migrating from Hive to Spark.• Implemented solutions processing over 1K daily Spark SQL queries, significantly enhancing data refresh rates and operational efficiency.Adobe Proof of Concept• Led the evaluation of Adobe Experience Platform, driving an estimated $200M in incremental annual bookings and achieving significant performance enhancements with an +80% lift in MQL conversion rate and +118% lift in SAO conversion rate.• Orchestrated efforts across 5 teams and 30+ individuals from LinkedIn and Adobe over 3 quarters, serving as the primary liaison and ensuring seamless integration and testing.• Addressed the complexities of integrating Adobe’s suite of services, rapidly becoming a domain expert. Designed and implemented crucial integration solutions that enabled the deployment of marketing use cases, significantly contributing to the project’s success.• Developed and led a set of tests to validate the scalability of Adobe features, delivering solutions to critical data integration challenges that unblocked key business metrics within 10 weeks.

Data Profiling System - A self-served data validation tool for the data warehouse. - Support metrics computation, data validation on large datasets. - Highly extensible for new dataset sources, validation rules, and statistic collectors. - Uses Spring, SpringMVC, Hibernate, ReactJS, Quartz scheduler, Hadoop, Spark, Azkaban.Integrate LinkedIn Segmentation and Targeting Toolset with the A/B testing platform TREX. - Design and Implement the segment-generation data pipeline that leverages Hadoop, Hive, Lucene Index, Azkaban, and Segment service. - Enhanced the application for segment definition and execution.

Build and operate a PB level Data Platform for AmazonClicks, which collects and processes billions of records per day and serves internal metrics, billing system and merchants’ real-time and scheduled reports.Raise HBase write/read speed by 15 times and make HBase serve 5 millions of requests on a 120 nodes cluster by optimizing cluster configuration based on deep analysis of data access pattern and re-designing client logic.Completely eliminated the downtime for merchants’ real-time reporting by building a seamless failover mechanism; enabled HBase fast recovery within 30 minutes by designing a reliable backup solution.Highly enhance Hadoop/HBase stability, efficiency and resolve complex issues, such as HBase read/write bottleneck, HBase Storage and Hadoop HDFS bottleneck, by diving deep and comprehensively analyzing HBase client/server, MapReduce, HDFS, Operating System and cluster Infrastructures.Lead the team to Hadoop 2 ecosystems and technologies by fulfilling and guiding the migration from Hadoop 1 systems, setting up production clusters for Hadoop 2.6.0, HBase 0.98, Pig 0.13, Zookeeper and Apache Drill.

Bring in thousands of merchants/vendors’ billing data to the current team by creating a reliable data workflow, which processes data worth millions of dollars revenue every year.Successfully integrate with Amazon Airstream in collecting and processing click, impression data by closely collaborating with multiple teams such as Amazon Retail Website and Amazon Iron Mountain Billing.Implemented and delivered Amazon Sponsored Products Campaign Performance Report, Suggested Bid Report and Device Type Report, which serve all Amazon Sponsored Products merchants.

Develop the BG’s (a social network benchmark) validation module using Map/Reduce on Hadoop Framework and MySQL.Write the client-side code and test MongoDB and Terrastore with BG. Test a cache manager similar to “memcached” for database system with several PhD students in Database Lab in USC.