Site Reliability Engineer
San Francisco, California, Us
Integrated service mesh technologies into Thumbtack's ecosystem, increasing visibility into application performance and providing traffic shaping features previously missing from the infrastructure. Built a control plane for Envoy Proxy that supported service discovery, rate limiting, and metric collection across the fleet of Thumbtack services. Debugged subtle networking issues while rolling out the service mesh in production.Created a new set of AWS virtual networks and supporting software in support of a company-wide infrastructure update. Designed the new setup with the goals of isolating environments (production, staging, etc) and with backwards-compatibility to facilitate migration to the new setup.Built tooling on top of PostgreSQL to provide scrubbed production data to developers in near real time. Leveraged ZFS, Docker, and AWS S3 to stream WAL data, snapshot complete servers, and run individual PG instances with minimal storage cost and in a highly scalable manner.Designed and implemented a canarying system for arbitrary applications, allowing new versions to be rolled out with fine-grained control over the traffic exposed to specific versions. Abstracted system through a script and set of daemons that allow engineers to deploy and rollout changes with simple commands. Integrated this system with the Thumbtack build process to enable automatic rollout of application versions. Customized Nginx configurations to allow for dynamic routing of traffic at the individual request level.Prototyped and deployed a new monitoring and alerting infrastructure. Evaluated multiple technologies and compared performance of production-level workloads, providing evidence to support the final decisions. Codified large parts of the monitoring pipeline, helping to ensure changes are correct and reliable through automated testing and manual reviews. Created standards and practices for recording and analysing metric data.