Downtime: When IT Goes Wrong

Got something to say to this podcast? Send a voicemail, and you might hear yourself on the next episode!

In June 2017 a poisoned update to a small Ukrainian accounting program unleashed NotPetya, a cyber weapon the US, UK and Australian governments later attributed to Russia. Within hours it had wiped the computers of the world's biggest container shipping company, A.P. Moller-Maersk, stopped chocolate production at Cadbury's Hobart factory and knocked a global law firm's phones and email offline. This is the story of how Maersk rebuilt roughly 4,000 servers and 45,000 PCs in about ten days, why its whole recovery hinged on one computer that happened to be switched off in Ghana, and what a cost of US$250 to 300 million teaches a 20-person Australian business.

You'll come away understanding why your backups need at least one copy that malware simply cannot reach, and why you must test restoring them; why the system that controls your logins is the first thing you need back and deserves its own written, rehearsed recovery plan; why prompt patching matters but is not enough on its own; how separating your network and limiting administrator accounts stops one infected laptop becoming a company-wide disaster; why every piece of software that updates itself is part of your security perimeter; and why a printed, paper-based way to keep trading for a week is one of the cheapest insurance policies you can buy.

A practical first step: this week, ask whoever looks after your IT when a full restore of your most important system was last tested, how long it took, and whether a copy of your backups is kept offline or locked against changes.

Downtime is brought to you by Complete Uptime Solutions, managed IT and cyber security for Australian businesses. Get a free IT risk snapshot at completeuptimesolutions.com.au or call 02 8214 4612.

Sources: Wired (Andy Greenberg, "The Untold Story of NotPetya"); A.P. Moller-Maersk interim reports and investor statements; Australian Government statement attributing NotPetya to Russia (February 2018); UK National Cyber Security Centre and the White House; ABC, iTWire and Fortune reporting on Cadbury and DLA Piper; Australian Cyber Security Centre Essential Eight.

Got something to say to this podcast? Send a voicemail, and you might hear yourself on the next episode!

On Friday 19 July 2024, a routine content update from security company CrowdStrike crashed about 8.5 million Windows computers into blue screens. Airlines, hospitals, banks, broadcasters and supermarket checkouts across Australia and the world stopped mid-afternoon, and many organisations spent days fixing machines one at a time. This episode tells the story, explains the root cause in plain English, and follows the money, including Delta Air Lines' half-billion-dollar fight with CrowdStrike.

You'll come away understanding why a security tool can take down every PC at once and why staged rollouts matter for the software you buy and the updates you push yourself; why your BitLocker recovery keys need to be reachable when every computer in the office is down; how to keep trading when your point-of-sale and booking systems are dark, with cash, a standalone EFTPOS terminal and paper fallbacks; why your incident plan needs a "count the machines and the hands" step and a scam warning for staff; and what your supplier contracts and insurance actually cover when the failure isn't a cyber attack.

A practical first step: this week, ask whoever manages your IT to show you, from a phone, the BitLocker recovery key for one of your office PCs. If they can't do it in five minutes, fix that first.

Downtime is brought to you by Complete Uptime Solutions, managed IT and cyber security for Australian businesses. Get a free IT risk snapshot at completeuptimesolutions.com.au or call 02 8214 4612.

Sources: CrowdStrike Preliminary Post Incident Review and Root Cause Analysis; Microsoft; ABC News; SBS News; Delta Air Lines filings via Fortune and Bloomberg; Parametrix.