- Repairing the bad CrowdStrike update likely cost more than $700 million, analysts said
- Delta's CEO said the company lost "half a billion dollars" in revenue thanks to cancellations caused by the issue
- Millions more was likely flushed down the tubes in the form of lost productivity
It was the software update felt round the world. The chaos caused by CrowdStrike’s recent fumble instantly made the company infamous, to the point that even my local Starbucks barista knew the cybersecurity company by name. But forget, for a moment, the damage to CrowdStrike’s previously sterling reputation. Let’s talk money.
There are a few different ways to size up the outage’s financial impact. The first, and probably most straightforward, is to look at CrowdStrike’s stock. As Futuriom’s Scott Raynovich noted, the value of CrowdStrike's shares fell dramatically in the wake of the incident. Raynovich put the lost value at 50%, but by Fierce's calculations the share price fell just over 30% from $377.37 on July 15 to $263.91 on July 22.
The same couldn’t be said for Microsoft, whose Office 365-powered machines were bricked by the bungled update. As investor site The Motley Fool pointed out, Microsoft’s stock barely budged (and, in fact, was more negatively impacted this week by — of all things — an earnings beat that was too narrow for investors’ liking).
Counting the cost
And what about the cost of repairing all those bricked machines? Well, Jack Gold, founder and principal at J. Gold Associates, crunched the numbers.
Using an estimated cost of $55 per hour to repair each machine, with a likely repair time of 1.5 hours, Gold figured “each machine on average would cost $82.50 to fix.” Multiply that by the 8.5 million machines Microsoft said were impacted and Gold said fixing the CrowdStrike code issue alone likely cost at least $701 million. And that doesn’t include lost revenues from service outages resulting from the code issue.
While that figure is much harder to size up, comments made by Delta’s CEO this week hint at the scale of the losses suffered. In an interview with CNBC, Ed Bastian said the incident cost his company “half a billion dollars.”
“It was terrible,” Bastian said, adding “we got hit the hardest.”
In response to questions from Fierce, Gold said he was actually surprised Delta’s costs were so low.
“Any flight delays or cancellations are very expensive, not to mention having to pay people to stay over when you cancel them and feed them, etc.,” he noted.
Gold said while each business’ losses were likely different, it is possible to come up with a rough number for “lost productivity.” If you assume each employee with a brick machine has an annual pay rate of $80,000, then each hour of lost productivity costs the business $28. And that’s probably a low estimate, he said.
“If you have 10K employees that are offline for an hour, that’s a lot of money. And if they’re offline for multiple hours, it all adds up, not to mention that $80K is pretty low for knowledge workers. So, lost productivity costs were pretty high,” Gold explained.
And again, that doesn’t include what Gold classified as “longer term and invisible effects.”
“If 10% of Delta customers decide they will never fly on Delta again, what does that cost them in lost revenue? It’s pretty expensive,” he said.
Lessons learned
As Raynovich put it, the outage taught enterprises worldwide a hard lesson they should already have learned: don’t put all your eggs in one basket.
“CrowdStrike had become an industry darling, with a lofty stock price and pundits slathering over its business success,” he wrote. “In the end, it's just another software company that demonstrated you are one mistake away from failure.”
He added that IT organizations need to reexamine vendor concentration levels, examine the CI/CD process and establish protocols to “test software updates from vendors and focus on phased rollouts, rather than trusting the vendor completely.”
But Gold, in contrast, argued that certain tech providers have become like electricity providers. They’re not just companies you can walk away from, even after such a massive outage.
“Yes this was a costly disaster, but I can’t see everyone switching. It would be very costly and highly disruptive, and in some cases impossible,” he said. “And what would they switch to? Apple? It has its own issues. Unix? No real user interface.”
Gold concluded: “I think the real issue is learning from this mistake and finding ways not to have it happen again. Crowdstrike did an awful job mitigating this before the fact—that’s a big part of the lesson here.
This story has been updated to correct a line about CrowdStrike's share price drop.