I founded the EMEA Reliability team to find and solve the hardest problems in HubSpot EMEA 500-person engineering team. I identified single points of failure across all HubSpot APIs and found consensus with HubSpot senior engineering leadership on the new higher standard that all critical services should meet.I lead the European Reliability team of 3 engineers to implement changes resulting from single points of failure findings including supporting automatic database failovers, adding redundancy storage, tuning timeouts for fast failure and recovery, tighter alerting thresholds, better caching, and more. I also embedded with 3 different engineering teams over the course of 7 months to help them solve complex aspects of distributed architecture at scale and equip them to respond better to reliability issues in the future. Completed projects include: - Email Sending Infrastructure - Modernized Email DNS - built a new API that's simpler to use, debug, and maintain. Previous API relied on specialized caches and technologies that nobody could understand. - Assisted Buying Experience - reduced API latency from 30 to 2 seconds and removed race conditions and timeouts. Improved dev productivity by decreasing build times from 15 to 5 minutes. - Notifications Platform - reduced data set and migrated to sharded MySQL. Helped reduce the amount of data storage needed by 70% to take pressure off an overwhelmed data ingestion pipeline, which allowed the team to shard the database and remove the problem.On top of all this, I lead dozens of high and medium severity incidents as a reliability expert, helped identify root causes and remediate, and participated in hundreds of incident postmortems.

- Transformed, modernized, and simplified the entire auth stack (running in 2500 micro-services across all of HubSpot), making systems more reliable and performant. - Identified that HubSpot was not prepared to meet Google’s new permissions guidelines by the deadline and formed an emergency team of 8 to modify how HubSpot interfaces with Google. Rewrote authentication and permissions APIs in 3 weeks, collaborating with 12 different engineering teams, along with Security, Finance, and Legal. Successfully rolled out to 100k customers with only 3 support cases. - Added progressive scoping to the OAuth system for integrators and led work to migrate the entire product feature flags framework.- Seamlessly migrated our integrators from OAuth1 to OAuth2, moving 150k customers with only 4 support cases. - Added support for 2FA and SSO to HubSpot login, which also required rewriting a large part of cookiegeneration.- Built a slack bot, now used across all of HubSpot, to automatically answer common questions in public Slack channels. This reduced questions requiring team engagement from 70 a week to 10.I then formed the App Platform team to apply learnings and solutions from the Auth team to scale and improve performance and reliability of core platform services (managing users and accounts, UI bootstrapping APIs). - Rewrote core APIs, paid off 10 years of technical debt, and seamlessly migrated 200 teams to new API versions.- Enabled HubSpot's EU datacenter by planning a multi-region strategy for a product group of 7 teams, solving technical challenges to allow users to exist in multiple regions and seamlessly change from one to the other without performance degradation.- Partnered with SQL infra team to implement global replication and become first adopters of the technology.- I operated as an unofficial interim director/architect for 30-40 engineers when our director left suddenly.