Last week saw a truly epic outage to Microsoft 365, with a huge number of users unable to send and receive emails. The company has now admitted that at the worst point, more than 80% of its servers were unresponsive.
Microsoft has now explained what went wrong, stating that the mistake that led to the failure was actually made all the way back in July without being detected …
The company said at the time that it was a technical issue and not a cyber attack, and that it was working hard to fix the problem, admitting that it was taking longer than expected. Microsoft also denied that it was related to the thousands of layoffs announced the previous month.
Users were understandably unimpressed at the scale of the outage and the time taken to fix it, although meme creators were having fun.
Microsoft’s explanation – two factors combined
Platform head Charles Lamanna admitted the severity of the problem.
In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.
He said it resulted from a combination of two issues. The company maintains two independent network gateway systems, one of which failed, meaning that the remaining gateway took the full load. That wouldn’t have been a problem on its own, Microsoft stating that each one has enough capacity to handle all traffic. However, a mistake made back in July led to a cascading failure.
A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers […]
The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.
In other words, the update wasn’t tested under the conditions in which it was most likely to fail.
9to5Mac’s Take
While the company claims that the incident is unrelated to laying off staff and replacing them with AI systems, I find myself extremely skeptical. It seems inconceivable that a company of Microsoft’s size could make what appears to be a schoolboy error of this magnitude unless there were insufficient engineers working on the update. Please share your own thoughts in the comments.
You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel
