- The FCC's report on AT&T's major outage said that a network update caused the outage
- The incident lasted 12 hours overall and hit millions of customers
- AT&T has implemented technical controls in its network and greater peer reviews to ensure it doesn't happen again
Imagine being an employee running a update on a cellular network and taking down the entire nationwide service for hours. That’s what the Federal Communications Commission (FCC) said happened to AT&T in the early hours of February 22, 2024.
“AT&T Mobility implemented a network change with an equipment configuration error on Thursday, February 22, 2024 at 2:42 AM," the agency said in a new report on the outage. “Just three minutes later, the nationwide outage began.”
For those who don't recall, the outage took place just before Mobile World Congress Barcelona, leaving AT&T the subject of plenty of talk at the show. The FCC opened a formal investigation into the incident on March 8. The New York State Attorney General's office also opened a probe, but does not appear to have published an incident report yet.
The configuration error caused the AT&T network to enter ‘protect mode’ to prevent impact to other services, disconnecting all devices from the network and prompting a loss of voice and 5G data service for all wireless users. “It took close to two hours to roll back the network change,” the FCC reported. “Full service restoration, however, took at least 12 hours because AT&T Mobility’s device registration systems were overwhelmed with the high volume of requests for re-registration onto the network.”
The outage left millions of people without wireless service during some – or all – of its duration. Users were unable to contact family, employers, emergency services or connect to the internet.
The outage affected users in all 50 states as well as Washington, D.C., Puerto Rico and the U.S. Virgin Islands. All voice and 5G data services for AT&T wireless customers were unavailable, affecting more than 125 million devices, blocking more than 92 million voice calls and preventing more than 25,000 calls to 911 call centers, the FCC said.
This outage also affected AT&T’s FirstNet emergency responder network. AT&T prioritized the restoration of FirstNet before other services, the FCC noted, but from 2.45 a.m. to 5 a.m. FirstNet services were unavailable.
The outage also impacted T-Mobile, Verizon and other operator’s users, the agency said. “Voice and 5G data services were unavailable to customers of other wireless providers that regularly use or were roaming on AT&T’s network,” it noted.
AT&T suffered another hiccup in June, when an interoperability issue between carriers led to another outage for customers.
Outage aftermath
“Within 48 hours of the outage, AT&T implemented additional technical controls in its network,” the FCC said. “This included scanning the network for any network elements lacking the controls that would have prevented the outage, and promptly putting those controls in place,” it added.
AT&T has also implemented additional steps for peer review and adopted procedures to ensure that maintenance work cannot take place without confirmation that required peer reviews have been completed.
Still, it seems enforcement action against AT&T could be on the table. The FCC said it referred the matter to its Enforcement Bureau to address potential violations of parts 4 and 9 of the Commission’s rules.