Unusual Ransomware Recovery
How to Survive a Ransomware Attack Without Paying the Ransom
Norsk Hydro used faxes, Post-its, and old PCs to beat cybercriminals.
https://teakdoor.com/images/imported/2020/07/443.jpg
Michael Hammer, Norsk Hydro’s plant manager in Cressona, Pa.
PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
By William Turton
July 23, 2020, 11:01 AM GMT+7
At around midnight Oslo time on March 19, 2019, computers owned by Norsk Hydro ASA, a large aluminum manufacturer, started encrypting files and going offline en masse. It took two hours before a worker at its operations center in Hungary realized what was happening. He followed a scripted security procedure and took the company’s entire network offline—including its website, email system, payroll, and everything else. By then, a lot of damage was already done. Five hundred of Hydro’s servers and 2,700 of its PCs had been rendered useless, and a ransom note was flashing on employees’ computer screens.
“Greetings!” the note began. “There was a significant flaw in the security system of your company. You should be thankful the flaw was exploited by serious people and not some rookies. They would have damaged all your data by mistake or for fun.” The message instructed recipients to write to an email address to discuss an unspecified payment, which would have to be made in Bitcoin; in exchange, the hackers would provide an encryption key to reverse the damage.
Like most other large multinationals, Hydro had been at least aware of the possibility of attack. It had a cyber insurance policy, and it had tested its networks with “white hat” hackers—security consultants who attempt to break into a system to check its defenses. “I wouldn’t say we could keep the NSA out,” says Chief Information Officer Jo De Vliegher. “But we were a company with all the normal security in place.”
https://teakdoor.com/images/imported/2020/07/444.jpg
De Vliegher COURTESY: NORSK HYDRO
It wasn’t enough. Some 35,000 employees were locked out of the company’s network, and Hydro had to shut down several manufacturing plants in Europe and the U.S. The ones still operating had to figure out how to do so without any computers. In the end, the attack would cost the company more than $60 million—way more than the $3.6 million the insurance policy has paid out so far, according to an earnings report. It was, according to the prosecutor investigating the breach, the worst cyberattack in Norway’s history.
Despite all this, Hydro never considered paying the ransom, because the anonymous hackers could have just taken their Bitcoin and disappeared. Even if they’d provided the key—and even if the key worked—it would have sent a message that Hydro was an easy mark, leading to future attacks and more extortion.
Instead, De Vliegher oversaw a fitful recovery from the attack, improvising with ancient PCs, fax machines, Post-it notes, and all manner of other analog technology. The response illustrates the painful reality that security consultants and law enforcement officials often bring up: Even when you do everything you can to protect yourself from a cyberattack, a determined adversary will almost always be able to wreak havoc. In other words, it’s less a question of how to stop hackers from breaking in than how to best survive the inevitable damage.
On the night of the attack, De Vliegher had just landed in Belém, Brazil, where Hydro has a large presence. As soon as he heard computers had been encrypted, he took the first flight home. By the time he made it back to Hydro’s corporate headquarters in Oslo, a team of five specialists from Microsoft Corp. was there, working to diagnose the problem and figure out how to restore the company’s data. Employees had taped handwritten notes to the doors warning others not to turn on any phones connected to the company network.
Hydro needed to alert customers, suppliers, employees, and investors, but the company’s website was down. So at 9:42 a.m. the day after the hack, an employee on the communications team used his personal cellphone to make a post on the company’s Facebook page: “Hydro is currently under cyber attack. Updates regarding the situation will be posted on Facebook.”
Next, Hydro had to make sure employees got paid. Banks were refusing to communicate digitally with the company, fearing that whatever had infected its network would spread to them next. Payday in Brazil was two days away, and 5,000 employees there were expecting a check. De Vliegher came up with a solution: He copied the previous month’s paychecks from an external payroll system, removing the employees who’d been fired or quit in the meantime. “It was about 90% accurate,” he says.
https://teakdoor.com/images/imported/2020/07/445.jpg
Shipping and receiving in Cressona. PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
Of all the many operations Hydro has around the world, from the bauxite mines in Brazil to the hydroelectric power plants in Norway (hence the name), the damage was worst in Cressona, Pa., where the company operates its largest aluminum plant. The Cressona facility was built by the U.S. government during World War II to make aluminum for weapons; it has a sawtooth roof that was designed to confuse enemy bombers into thinking they were looking at ripples on a lake. The plant is run by Michael Hammer, who started there 25 years ago in accounting and stayed on as it was passed among different owners. (Hydro acquired Cressona in 2017.)
It was dinnertime in Pennsylvania on March 18 when Hammer got a call from Hydro’s vice president for risk management. “Get your folks to the plant,” he remembers the VP saying. “Print out as much stuff as you possibly can before they start pulling the plug on the servers.” Hammer had experienced brief outages before. Maybe someone down the road ran their car into a power line, he thought, figuring the plant would come back online in a few hours.
He knew it was bad as soon as he arrived and saw workers frantically unplugging computers. Then he read the ransom note. “I didn’t even know what the hell Bitcoin was,” he says.
https://teakdoor.com/images/imported/2020/07/446.jpg
The Cressona plant. PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
Under normal circumstances, his plant employs 1,180 people, runs 24/7, and produces more than 2.6 million pounds of finished aluminum a year. Walking through it today, you can feel the heat from the furnaces where recycled metal is melted down and reformed into large cylinders. These are heated and pushed through 60-pound circular dies, transforming them into components for such products as window frames and flooring. Imagine pushing Play-Doh through a cookie cutter. Customers include Tesla Inc. and Ford Motor Co.
This kind of manufacturing predates computers, but computers have made it much more complex. Hydro has more than 50,000 dies, and it uses software to keep track of what’s being made and to tell employees which die to pick off the shelf. Without access to customer orders, technicians had no idea what to make. Hydro employees began calling customers, asking them to text or send orders to personal email accounts. With the corporate email system down, plant staff traded phone numbers and communicated by group text.
https://teakdoor.com/images/imported/2020/07/447.jpg
A temporary war room in Cressona. PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
As the orders started to trickle in, the only way for people on the plant floor to know what to do was by reading off a paper copy of each order. Luckily the plant had a bunch of old computers in storage, which Hammer set up in a war room to print the forms. “We went over to Staples, and we pretty much cleaned them out of printers and paper and cartridges,” he says. Salespeople, whose computers were also hacked, had nothing to do, so Hammer had them strap on safety gear and run paper orders to workers on the plant floor.
https://teakdoor.com/images/imported/2020/07/448.jpg
Printed work orders kept the factory running while the network was down. PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
For the first week, Hammer lived at the plant, occasionally taking naps on a couch in his office. Losing access to Hydro’s network also meant he wasn’t able to pay his monthly bills to suppliers, and they were calling to ask where their money was. So he pulled an old fax machine out of a closet and asked suppliers to fax payment details, which he then forwarded to Hydro’s bank. The suppliers who still had fax machines lying around got paid first.
Hammer is still searching for answers as to who could have attacked his plant and gotten away with it. “It was a lot of manual stuff, a lot of long hours, a lot of long days,” he says. “And that pain was injected by an evil person. It was a terrorist basically. And what made it worse is it was nameless, faceless. You don’t know where it came from, how it got there.”
https://teakdoor.com/images/imported/2020/07/449.jpg
Hammer PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
Nobody has figured out who attacked Hydro, but signs point toward an organized cybercrime group operating with impunity somewhere in Eastern Europe. The group made headlines last year for hacking point-of-sale systems to steal credit card numbers. Known to security researchers as FIN6, it’s often extracted Bitcoin ransoms in the hundreds of thousands of dollars. “Fin” is short for “financially motivated,” to differentiate the gang from military hacking units affiliated with countries that have active cyberweapons programs, including China, North Korea, Russia, and the U.S.
FIN6’s signature weapon is a virus called LockerGoga, named after one of the files buried in its malware. There are dozens of variants of the software, and Hydro thinks the attackers deployed more than one within its network, making it harder to expunge from the company’s systems.
Ransomware hackers generally penetrate computers more or less at random, then use a self-propagating software program—a worm—to work their way deeper into the corporate network. But in Hydro’s case, the attackers gained access by hijacking a legitimate email from an Italian customer. The customer had attached a file, which the hackers modified. When the file was opened, on Dec. 5, it executed malicious code, allowing the invaders access to the entire network. They waited until March to launch their attack. The company doesn’t know if the hackers first compromised the customer or if the message was intercepted and changed in transit.
Hydro wasn’t the first industrial company to be hit by the LockerGoga virus. A French engineering company, Altran Technologies SA, was struck in January 2019. Later that year, U.S. chemical companies Hexion Inc. and Momentive Performance Materials Inc. received copies. Large industrial companies aren’t conventional ransomware targets, leading some computer security researchers to wonder if the attacks were about sabotage rather than greed.
In addition to encrypting Hydro’s computers, the virus changed the password of every administrator account, logged those accounts out, then restarted each computer, making it harder for employees to even see the ransom note—which didn’t include a specific demand for money, or even the address of a Bitcoin wallet. There was just an email address. Of course, these idiosyncrasies could have been dreamed up by FIN6 to make Norsk executives feel more vulnerable, says Charles Carmakal, senior vice president for cyber[at]security firm Mandiant. Norsk says there’s no evidence the hackers wanted anything other than money.
Investigators at Kripos, Norway’s equivalent of the FBI, and Europol, the EU’s law enforcement agency, are still sifting through terabytes of data from the hack. They’re not especially optimistic about making an arrest. Cyber[at]crime groups use encrypted apps and take payment in cryptocurrency, making traditional policing tools, such as wiretaps and search warrants, useless. On top of that, the cross-border nature of crime creates mountains of paperwork to retrieve evidence that may be stored on servers in another country. “The criminals can communicate freely without law enforcement being able to read what they are saying,” says Knut Van Jostein, the prosecutor leading the investigation.
Back at Hydro’s headquarters, the emergency response team spent weeks locked inside a conference room as they rebuilt the entire network from scratch. They were para[at]noid about any further intrusions, so even the cleaning staff was barred from entering. De Vliegher says the room got very messy. “This is the most secure room we have, so we don’t want anyone to leave whatever spy pens and microphones and stuff behind,” he says in an interview in Oslo.
https://teakdoor.com/images/imported/2020/07/450.jpg
Signs reminded employees not to use devices connected to the network. PHOTOGRAPHER: WILLIAM MEBANE FOR BLOOMBERG BUSINESSWEEK
Recovery meant creating a safe zone of computers that definitely didn’t have the virus and slowly moving other machines that had been verified as clean over to the new network. Progress was slow. Three weeks after the attack, Hydro had a total of four functioning PCs in all of the U.S.
Employees in France set up a make-shift assembly line to build new, noninfected PCs, and created a sort of bucket brigade to transport PCs across Europe. Workers drove to a gas station in the middle of the country to swap infected computers for clean ones. At a plant in Magnor, east of Oslo, pensioners who lived nearby came out of retirement to help with printing and sorting orders.
Hydro executives are grateful the loss was just $60 million. In the darkest days following the hack, some feared they’d fall so far behind on orders it would sink the entire company. “We came out of it stronger because of all the 35,000 people that worked overtime, weekends, changed jobs. Nobody complained,” De Vliegher says. “But in a company where that willingness is not there, it’s lethal.”
Things were mostly back to normal when a Bloomberg Businessweek reporter visited last September, but the company still hadn’t fully recovered. In Magnor, employees had lost access to the software that runs its production line. Luckily, a similar plant in Denmark was spared, and an employee there sent a copy of the program on a flash drive. The staff electrician in Magnor, who moonlights as an IT support guy, figured out how to install thenew copy. The software works well enough, though it’s all in Danish.
