GitHub’s Nearly 8-Hour Outage Traced to an Autoscaling Blind Spot and a VS Code Retry Storm

GitHub’s Aug. 17 outage ran 7 hours and 47 minutes, from 13:28 to 21:15 UTC. It caused errors across Issues, pull requests, APIs, Actions and Copilot, and authentication failed with them. “If you were trying to ship software that day, we let you down,” CTO Vladimir Fedorov wrote in a post on GitHub’s blog. It was GitHub’s second major incident of the month, after an Actions failure on Aug. 6.

What broke

GitHub’s root-cause analysis, summarized by The Register, says traffic reached a new peak and an Istio sidecar in the Central US data center hit its concurrency limit. That saturated network capacity on the load balancers. Autoscaling didn’t help, because the policy watched the host service and not the sidecar’s concurrency limit. In GitHub’s words, “The problem was worsened by optimistic retry logic which overloaded internal load balancers.”

Copilot made recovery harder. “Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service,” GitHub wrote. Engineers cut gateway retries with a code change and had the load balancers answer Copilot Token Service requests with HTTP 403 errors. Scraping attacks on codeload endpoints added to the load. Most services were back by 16:36 UTC and Actions by 18:03 UTC, but the token service took until 21:02 UTC.

Fedorov said that “neither outage was caused by a code or configuration change.” He called both “capacity failures at their core.” Monthly commits have grown from 1.4 billion in April to 2.9 billion.

The fixes

GitHub said it will correct its autoscaling policies, audit Istio concurrency settings and deal with the VS Code behavior. Fedorov added that GitHub will apply consistent retry limits, retry budgets and variable timeouts between services, review lower-priority CPU and memory alerts that could warn of a spike, and separate critical systems from shared dependencies.

Since its reliability push began, GitHub has added more than 3 million CPU cores and 120 petabytes of high-speed storage. Azure now serves about 58% of platform load and half of all Git operations, up from 12% of load in May. “Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations,” Fedorov wrote, adding that it will roll out first to the largest monorepos.

Some customers are looking elsewhere. CloudBees CEO Moritz Plassnig wrote on LinkedIn that “Cursor, OpenAI and several smaller startups are already building competitive solutions,” and The Register reported that paying customers complained on social media that the service is falling short.