
Microsoft outage caused 80% of servers to go down – here’s what happened

Last week, Microsoft 365 experienced a truly epic outage, with large numbers of users unable to send and receive emails. The company has now admitted that at worst, more than 80% of its servers became unresponsive.
Microsoft has now explained what went wrong, claiming that the error that led to the failure was actually made as early as July without being detected…
The company said at the time that it was a technical issue and not a cyberattack, and that it was working hard to resolve the problem, admitting that it was taking longer than expected. Microsoft also denied that this was linked to the thousands of layoffs announced the previous month.
Users were understandably unimpressed by the scale of the outage and the time it took to repair it, although the meme creators were having fun.
Microsoft’s explanation – two factors combined
The head of the platform, Charles Lamanna, recognized the seriousness of the problem.
In less than two hours, the previously unknown fault was detected throughout our entire fleet and rendered it unresponsive. At its lowest, more than 80% of servers were unresponsive.
He said it was the result of a combination of two problems. The company operates two independent network gateway systems, one of which failed, meaning the remaining gateway took all the load. This wouldn’t have been a problem in itself, with Microsoft saying everyone has enough capacity to handle all the traffic. However, a mistake made in July led to a cascading failure.
A latent defect caused an overload of servers which no longer responded under certain load conditions. This was due to a lock conflict within an internal process that made the machine unavailable. This defect is the result of a patch prepared specifically for Microsoft 365 network gateway servers (…)
The first behavior change began rolling out in July this year. The update was deployed safely and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed around the world without impact. Our normal load testing did not detect the latent issue due to the different traffic and SKU combinations used in our test labs. The contention problem is worse on machines with more cores and newer hardware; while our load tests tended to focus on our weaker hardware SKUs with fewer cores.
In other words, the update was not tested under the conditions in which it was most likely to fail.
Take from 9to5Mac
Even though the company claims that the incident had nothing to do with laying off staff and replacing them with AI systems, I find myself extremely skeptical. It seems inconceivable that a company the size of Microsoft could make what appears to be a schoolboy error of this magnitude unless there weren’t enough engineers working on the update. Please share your own thoughts in the comments.


FTC: We use automatic, revenue-generating affiliate links. More.

Gn bussni