
Inside Microsoft’s giant Outlook outage
Last Thursday, at 9 a.m., everything seemed normal. Microsoft employees were arriving on campus, ready for a regular day of emails and meetings. But just an hour later, one of the worst outages in Microsoft history began to occur, leaving employees, businesses, and many governments without access to Outlook emails for nearly 24 hours.
It all started in Switzerland, according to a source close to the matter. Exchange Online servers that power Outlook have started going offline. The problem quickly spread to Mexico, and then to other countries around the world, as Microsoft’s failover systems attempted to move email traffic elsewhere. Within hours, more than 80% of Exchange servers were no longer responding.
Sources tell me that Microsoft marked the outage as “sev0” in its internal incident tracking system, the highest level, indicating a catastrophic level of service interruption. Hundreds of engineers began trying to figure out what went wrong and how to restore service. It’s not uncommon for servers to go down in certain circumstances, but Microsoft has various systems that are supposed to route traffic to healthy servers in order to keep the system as a whole online. In this case, the failover systems did their job, but they also propagated the problem.
From the first hours of the outage, engineers were analyzing crash reports, trace logs, and kernel dumps to try to find clues to the giant failure. Some servers had entered maintenance mode and became unresponsive due to unusually high CPU loads. Engineers attempted to bring the servers back online, but they were almost immediately overwhelmed by traffic waiting to access the servers.
Exchange servers start and run an operating system over a network (Preboot Execution Environment), a process that allows Microsoft support staff and engineers to manage operating system updates remotely. I understand that the sudden burst of traffic from the rebooting servers was overwhelming the systems monitoring them, causing a form of reboot loop in which they unsuccessfully tried to get back online.
As engineers continued to look for the root cause, Microsoft even struggled to return the servers to a healthy state using its usual processes of rolling back recent changes. As the outside world began to wonder whether this outage was related to a cyberattack or sloppy AI coding, Microsoft engineers were already studying potential attack vectors and reaching out to DDoS specialists to understand if it could be an attack.
I’m told that crash reports and logs quickly ruled out the possibility of an attack, and after recent restores failed, Microsoft engineers began looking more closely at the servers’ networking stack and discovered an unusual bug. This was unusual because the bug persisted on Microsoft 365 network gateway servers since July without any impact on the servers.
“A latent defect caused servers to become overloaded and unresponsive under certain load conditions,” wrote Charles Lamannathe head of Microsoft’s Copilot, Agents and Platform (CAP) team, in an article on X. The teams behind Office, Outlook and Microsoft Teams all began reporting to Lamanna last month after the head of Office and Teams, Ryan Roslanskyleft the company.
Microsoft’s preliminary post-incident report describes the bug as affecting some Windows servers “due to a lock conflict within an internal Windows process.” The bug does not affect standard versions of Windows because it is the result of a patch prepared specifically for Microsoft 365 network gateway servers.
Microsoft didn’t find the bug before this major outage, and admits it was due to the combination of traffic and the hardware used in its test labs. “The contention problem is worse on machines with more cores and newer hardware, whereas our load tests tended to focus on weaker hardware with fewer cores,” Lamanna wrote.
The bug became evident due to two other incidents that increased load and reduced server capacity. Fifteen hours before this Outlook outage began, Microsoft Teams moved from an alternate network gateway to the primary gateway supporting Exchange Online. “This increased the load on the network gateway and changed its traffic patterns,” Lamanna wrote. Normally, this wouldn’t have been a problem, but at the same time, approximately 14% of that gateway’s server capacity was “out of service due to an unrelated issue the day before.”
The trifecta of a bug, Microsoft Teams failover, and reduced server capacity all caused the network gateway to trigger the issue that had been dormant since July. It took Microsoft nearly 24 hours to stabilize its Exchange servers again, and I’m told that overlapping teams trying to resolve problems sometimes makes coordination difficult.
Microsoft engineers had a telephone bridge the whole time, and at one point it reached the maximum of 1,000 participants. This shows you how many teams were involved in trying to find the root cause of this huge outage, which started at peak traffic in the United States.
“Recovery took longer than usual because when the servers were brought back into service, they were immediately overwhelmed with waiting traffic and became unresponsive,” Lamanna wrote. “Our normally automated recovery processes were not designed to restore service from this general unresponsive state. Additionally, because the primary code defect was a latent bug from several months ago, our normal indicators of restoration, theft, and functionality were unable to quickly roll back the change.”
Microsoft is currently working on a final post-incident review which will be released later this week. Even without this comprehensive review, it’s clear that the company will need to invest more in its testing processes to ensure that when something like this happens again, it can detect the problem much earlier. Outages happen daily, but this type of cascading outage is a rare time for Microsoft to learn from its mistakes and ensure that the same sequence of events doesn’t bring down Outlook for millions of people again.
Notepad will return in December after next week’s Thanksgiving holiday.
I’m always eager to hear from readers, so feel free to leave a comment here, or you can reach me at notepad@theverge.com if you’d like to discuss anything else. If you’ve heard about any of Microsoft’s secret projects, you can reach me by email at notepad@theverge.com or talk to me confidentially on the Signal messaging app, where my name is tomwarren.01. I’m also Tomwarren on Telegram, if you prefer to chat there.
Thank you for subscribing Notepad.
Gn bussni