When the Redundancy Is a Rumor
I’ve been traveling, and I’m finally catching up on updates and what a week to catch up on. A nationwide ground stop, a day-long Microsoft 365 outage, and a run of supply-chain attacks that reached some of the biggest names on the internet. The upside of writing this a week out is that the facts have had time to settle, and the picture they form is worth your attention.
The Pattern I’m Watching
In 2003, the Northeast blackout took down power for roughly 50 million people across eight states and Ontario. The grid had redundancy. Utilities had failover plans. None of it mattered, because the failure was a cascade: one overloaded line in Ohio tripped, the automated response shifted load to adjacent lines, those lines tripped, and the cascade moved faster than any operator could intervene. The final report from the joint U.S.-Canada Power System Outage Task Force ran to 238 pages. The core finding was not that the utilities lacked redundancy. It was that nobody had tested what happened when multiple redundant components failed in sequence, under load, in the same window.
That finding was on the table for Microsoft 365 two weeks ago. Last Friday, it landed on the FAA.
On November 12, 2026, at 16:17 UTC, an overwhelmed core process caused Microsoft 365 servers to fail their automated health checks. Servers that failed health checks dropped out of rotation. The remaining infrastructure absorbed the load, failed its own health checks, and dropped out too. The cascade ran across Exchange Online, Teams, SharePoint, Copilot, Defender XDR, Intune, and Purview. Downdetector logged more than 130,000 Outlook problem reports at peak, per BigGo Finance’s incident analysis. The outage hit 74 countries. Recovery took nearly 24 hours. Microsoft ruled out a cyberattack and attributed the failure to a network capacity upgrade that surfaced under a subsequent network change, per WIRED.
After 30 years of watching infrastructure cycles, the pattern I keep seeing is this: every generation of “reliable” infrastructure eventually fails in a way that reveals the gap between documented redundancy and tested redundancy. The 2003 blackout exposed it in power. The 2011 AWS US-East-1 outage exposed it in cloud storage. The 2021 Facebook BGP withdrawal exposed it in global routing. Each time, the organizations that recovered fastest were not the ones with the most elaborate runbooks. They were the ones who had run a real drill against a real dependency going dark.
Microsoft 365 is load-bearing infrastructure for a significant share of the enterprise market. When Exchange Online goes down, email and calendar stop together. When Teams goes down, the communication layer that replaced your phone system goes with it. When Copilot goes down, the AI-assisted workflows your team built on top of it stop too. A Microsoft 365 outage does not just take away email. It takes away every workflow built on the assumption that these services would be available.
The recovery timeline adds a second lesson. Microsoft’s initial fix attempts failed. Restored servers became unresponsive again under traffic load, per Cyber Security News. The company had to deploy an alternative health-check method and build an automated process to detect and restart unresponsive servers mid-incident. That is not a fast recovery path. It is a 24-hour recovery path. Your team’s continuity plan needs to account for a full business day without Microsoft 365, not a 30-minute maintenance window.
The FAA Ground Stop: The Story Still Has Holes
Then Friday, November 13 happened. Telecom work at a single remote FAA radio site in El Dorado, Arkansas destabilized communications at 17 of the nation’s 22 en route air traffic control centers, and the FAA halted departures nationwide for more than four hours, per Al Jazeera.
Here is what has been established so far. L3Harris, the contractor that owns and operates the FAA’s telecommunications network, was migrating the El Dorado site from copper to fiber, and the work involved installing a new Juniper router, per AeroTime. Within about eight minutes of installation, the router began transmitting far too much data into the FAA’s network. The agency says connection attempts flooded the network, destabilizing aircraft communications and radar feeds, and it has since disabled a broadcast feature on the equipment, per TechTimes. Investigators have replicated the overload in a lab, per CNBC, but still cannot say whether the trigger was faulty hardware, a faulty installation, or a faulty configuration. HPE, which owns Juniper, says its devices “did not fail.”
I spent three decades around networks like this, and my first reaction on Friday was that the official framing did not add up. One site should not be able to take down a nationwide system. For the loss of El Dorado to sever communications at 17 of 22 centers, the network would have to be some hub-and-spoke design where a single remote site in Arkansas sits on the critical path for national egress. Nobody builds a safety-of-life network that way, and nothing suggests the FAA’s is built that way. So “telecom work at one site” is not an explanation. It is a location.
Here is the technical problem with the story as told. A national air traffic telecom network is a routed WAN, not one flat VLAN. On a routed network, a misbehaving device’s broadcast traffic dies at the first Layer 3 boundary. A broadcast storm can wreck its own site and nothing beyond it. For instability from one router in Arkansas to reach 17 of 22 centers, something had to carry it across the entire network, and the things that span a routed WAN are the control plane and whatever overlays ride on it. That points at routing-protocol propagation. BGP route churn is my lead suspect, with an IGP flooding storm or a multicast or Layer 2 overlay as the remaining candidates. The FAA saying it “disabled a broadcast feature” afterward is telling either way: either that feature was a routing or multicast mechanism that let one device’s chatter propagate network-wide, or the network extends Layer 2 across far more sites than anyone should be comfortable with in 2026. Both are findings the final report needs to address by name, and neither one is “we were swapping copper for fiber.”
Redundancy did not fail here. It was never in play. Redundant paths protect you against subtraction, a line cut, a site dark, a router dead. They do nothing against a device injecting garbage into the control plane, because every redundant path carries the garbage just as reliably as the traffic. That is the same family of failure as Facebook’s 2021 BGP withdrawal: the failure traveled over the very network that was supposed to contain it. I will update this read when the root-cause report lands.
The questions that still do not have good answers:
Why no fast failback? This was a hardware swap. The new router went bad within eight minutes. The old configuration existed that morning. Rolling back a failed cutover is the most basic move in any maintenance runbook, yet the national airspace was degraded for hours. Part of the answer may be that nobody could see the problem: during the event, L3Harris reportedly struggled to even locate where in its own network the failure was, per CBS News. You cannot roll back what you cannot find.
Who was watching? CNN reports the FAA had little oversight of the work. L3Harris notified the agency on November 9 that it would replace a router at El Dorado starting at 9 a.m. on Friday the 13th, and under a March 2025 FAA memo, certain contractor work could proceed on notification rather than approval. Sources told CNN flatly: “there was no monitoring,” per CNN.
Why that window? Every network maintenance culture I have ever worked in puts risky cutovers in the lowest-traffic window available, overnight, midweek, with engineers staged to fail back. This one started at 9 a.m. on a Friday, one of the heaviest travel days of the week, with the national system at full load. That is not a technical failure. That is a change-management failure, and it was decided before anyone touched a cable.
The FAA has paused further router installations while the investigation continues, per CNN, and Senate Democrats are pushing for independent testing before the remaining swaps proceed, per Yahoo News. Transportation Secretary Duffy has pointed at the “incredibly old” air traffic system, and he is not wrong that it is old. But the component that failed here was the new one. Modernization does not remove this class of risk. Done without monitoring, testing, and a rollback plan, it is this class of risk.
Meanwhile, in Seattle
The same week brought two supply-chain stories close to home for anyone with ties to the Pacific Northwest.
Expedia Group confirmed that attackers reached its systems through a compromised third-party data platform provider, per Skift. The extortion group calling itself n0n claims to hold more than 830 million loyalty-program records, 58 million traveler profiles, and 333 million consumer account records, per Security Affairs. Treat those numbers as the attacker’s marketing until Expedia’s investigation says otherwise; the roughly 500MB published so far shows real customer emails, loyalty details, and traveler profiles. Even if the full claim is inflated, a list of real email addresses tied to real bookings is a phishing kit waiting to happen.
Then on Monday, researchers disclosed a supply-chain attack on Optimizely, the A/B testing and personalization platform. Attackers abused a feature that lets customers inject their own JavaScript into pages, and malicious code was served through 188 snippets across 178 organizations, including CNN, the BBC, Zoom, Salesforce, GitLab, Equifax, and Starbucks’ Canadian site, per TechTarget and The Register. The payload hunted for cryptocurrency transfers and payment-card details. There is also an unverified claim by the same n0n group of a ransomware hit on Starbucks itself; neither Starbucks nor any major outlet has confirmed it, so I am filing that one under “watch, don’t repeat.”
Notice what connects these to the FAA and Microsoft stories. Expedia was not breached through Expedia. The 178 Optimizely victims did not get hacked; their vendor did. The FAA’s network did not fail; its contractor’s change did. Your dependency map does not stop at the services you chose. It extends to every vendor your vendors chose, and most continuity plans have never drawn that second circle.
The ChatGPT Wallet, Same Lesson
The ChatGPT Wallet, launched November 13, adds a different angle to the same underlying pattern. OpenAI’s “dots” agents can now initiate purchases on behalf of users, with per-transaction biometric approval required at launch, per Unite.AI. OpenAI has indicated that future versions will allow users to delegate spending under conditions, including spending limits, permitted categories, and time windows. That is a reasonable design. The risk is one this month keeps surfacing from different directions. Vendors ship the capability first. The governance layer, spend controls, audit trails, approval workflows, arrives later. Teams that adopt early are running without it.
The 2003 blackout produced a mandatory reliability standard called NERC CIP. It took a cascading failure affecting 50 million people to produce a requirement to test what you claimed was redundant. Enterprise IT does not have a NERC CIP equivalent for cloud dependencies, and as this month made clear, neither does the FAA’s contractor-run telecom network. That means the testing is optional, which means most teams skip it, which means the first real test is the incident itself.
The Bottom Line (No Jargon Edition)
The Microsoft 365 outage began November 12, 2026 at 16:17 UTC and affected Exchange Online, Teams, SharePoint, Copilot, Defender XDR, Intune, and Purview across 74 countries, per Cyber Security News. The root cause was a cascading server health-check failure triggered by a network capacity change. Recovery took nearly 24 hours. If your team has not reviewed its continuity plan for a full-day Microsoft 365 outage, this week is the time.
The FAA ground stop on November 13 came from a copper-to-fiber cutover at one contractor-run site in Arkansas, where a newly installed router flooded the shared network within eight minutes and destabilized 17 of 22 en route centers. A routed national network is not one flat VLAN, and broadcast storms stop at routed boundaries, so for one router’s misbehavior to spread that far, the propagation had to ride the control plane — a routing-protocol event, not a cabling problem. Redundancy against loss is not redundancy against noise, and most architectures only test the first kind.
The FAA work started at 9 a.m. on a Friday, was performed on notification rather than approval, and ran with no real-time monitoring, per CNN’s reporting. Before your next infrastructure cutover, ask three questions: who is watching it live, what is the rollback trigger, and why is it scheduled when it is scheduled? If any answer takes longer than a sentence, the change is not ready.
The cascade in the Microsoft outage happened because servers failing health checks shifted load to remaining servers, which then failed their own health checks. Microsoft’s initial recovery attempts failed when restored servers became unresponsive again under traffic, per Cyber Security News. Your failover plan should assume the first fix attempt fails. What is your procedure for hour two, hour six, hour twenty-four?
Expedia confirmed a breach through a third-party data platform, and 178 major brands served attacker-controlled JavaScript through Optimizely the same week. Your security perimeter includes every vendor your vendors use. If your vendor risk review only covers your direct contracts, it covers less than half your exposure.
OpenAI launched ChatGPT Wallet on November 13, 2026, letting its “dots” agents initiate purchases with per-transaction biometric approval, built with American Express, Mastercard, and Visa, per Unite.AI. Enterprise spend controls, audit trails, and policy enforcement are not in the current launch scope. Build those before you enable this for a team account.
Start this week by identifying which of your team’s workflows stopped entirely during the Microsoft outage, documenting what the manual fallback was and whether your team actually knew it, then scheduling a tabletop exercise for a full-day outage of your most load-bearing dependency before the end of the quarter. The exercise does not need to be long. It needs to answer one question. What does your team actually do when the thing you assumed would always be there is not?
If you find this useful, subscribe for free. Every issue lands Saturday morning. Subscribe to Tech with Darin
The Question Worth Sitting With
The 2003 blackout produced a mandatory testing requirement because voluntary testing had not happened at the scale needed. Twenty-three years later, a contractor swapped one router at one site on a Friday morning and grounded the national airspace, and a health-check change took Microsoft 365 away from 74 countries for a day. The tools to test failover exist. The runbooks exist, at least on paper. The gap is between having a plan and having a plan that has been run against a real dependency going dark, including the dependencies two vendors deep that you have never listed.
When your team last ran a real continuity drill, did it include the failure mode where a component misbehaves loudly instead of dying quietly? Leave your answer in a comment on the post, not as a reply to this email. Replies land in a private inbox where no other reader can see them, and this conversation is worth having in public.
— Darin

