Data Reliability Engineering

Regression + Integration testing framework

Facebook Containers team working on Antlir, a filesystem image builder which can reproducibly build, test, and run OS images for containers and hosts.

• Implemented a self-serviced account reactivation flow which increased account reactivations by ~600% (roughly 1500 reactivations per day) and decreased account-related support tickets by 73%.• Formulated and executed a migration plan for over 360 million live user images from legacy S3 bucket. Using NGINX's HTTP proxy and AWS Batch Operations, completed a full migration in 3 days with no customer impact.• Added "spare key" functionality to authentication flows using AWS Elasticache and device cookies, so that authenticated users can securely bypass Twitch's login throttling and are protected from denial of service (DoS) attacks.

• Spearheaded the initiative to analyze over 260 million entries in the site's users database, identify and purge obsolete records used to reserve logins, and migrate important users to a new reservations database.• Migrated GET channels endpoint, which handles over 2000 requests per second, from outdated Rails monolith to new Go-based service.• Enabled user-driven account reactivation by developing "un-deletion" endpoints for login/signup modules.

• Aided in the migration of existing software infrastructure from AWS to GCP (Google Kubernetes Engine) by designing and deploying custom Kubernetes/Helm implementations for multiple microservice-based projects. In doing so, I was able to increased error visibility, system scalability, and overall ease of deployment.• Bootstrapped a new staging environment system for developers (named Tesseract) using Docker, Kubernetes, Helm, and a Golang-based CLI. Tesseract allows for multiple staging environments running in parallel which alleviates contention in the development cycle. In addition, it reduces the initialization time of the staging environments from 15 minutes to 5.• Assisted in the automation of a new data processing/anonymization pipeline using Apache Beam and Google Cloud Dataflow. This new pipeline, which aggregates data from MongoDB and PostgreSQL sources in order to create anonymous seeding data for developers, resulted in the reduction of average seeding time from 60 minutes to 90 seconds (approximately 40x improved performance). • Superseded the current container building pipeline with an automated one using Google Container Registry/Builder integrated with Github repository push triggers. This, in turn, eliminated the need to manually trigger builds as well as rebuild existing images.

• Designed and implemented the back-end API (primarily Python+Flask) for features in Wizeline's Roadmap such as Object Linking, Item Ownership, Item List View, and Action Undo/Redo. The deliverance of these functionalities was mirrored by an 18% increase in product usage.• Automated the congregation of usage statistics for Wizeline's Roadmap product via data mining techniques applied to PostgreSQL databases (hosted via AWS RDS). Scraped other sources of event data such as Kibana/ElasticSearch, Intercom, and Jaco. The automation of this data collection and correlation process resulted in a 6000% increase in efficiency.• Re-created the structure of the local development environment to match Roadmap’s production instances using AWS ECR, Kubernetes (Minikube), Docker, and Apache Kafka+Zookeeper. This new local environment led to more consistent performance and took ⅙ the time to build compared to the old environment.• Built a code security test into Wizeline’s standard Jenkins Pipeline which was used by over 10 internal and external projects.