‘Unprecedented’ Google Cloud event wipes customer accounts and their backups

Buried in this week’s Google I/O news is one of the biggest failures in Google Cloud history. Google’s Amazon Web Services competitor accidentally deleted a huge customer account for no reason. UniSuper, an Australian pension fund with $135 billion worth of funds under management and 647,000 members, had its entire account wiped on Google Cloud, including all backups stored on the service. UniSuper thankfully had some backups with another provider and was able to recover the data, but according to UniSuper’s incident logs, the downtime began on May 2nd for his Full restoration of service did not occur until his May 15th.

UniSuper’s website is currently full of nightmare fuel that is a must-read for administrators about how this happened. first, wild page It was posted on May 8th under the title “Joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kuran.” The statement read: “Google Cloud CEO Thomas Kurian said the disruption was caused by an inadvertent misconfiguration during the provisioning of UniSuper’s private cloud service that ultimately resulted in the failure of UniSuper’s private cloud subscription. Google Cloud acknowledged that this was an isolated and “unique event” that had never occurred before at any of Google Cloud’s clients around the world. We have identified the events that led to this disruption and have taken steps to prevent it from happening again. . ”

The next section is titled “Why did the disorder last so long?” “UniSuper had redundancies in the two regions as protection against outages and losses. However, when UniSuper’s private cloud subscriptions were removed, deletions occurred across both of these regions,” the joint statement said. It has been stated. All cloud services maintain full backups, which are supposed to be for worst-case scenarios. Imagine a hacker taking over your server or the building containing your data collapsing. But no, actual The worst case scenario is that Google deletes your account, which means you lose all your backups as well. Google Cloud is supposed to have safeguards that don’t allow account deletion, but apparently none of them worked and the only option was a restore from another cloud provider (his Thank you UniSuper heroes).

UniSuper is Australian. ”retirement fund“The US equivalent is a 401(k), which is a retirement benefit paid by an employer as part of an employee’s salary. In Australia, all employers are required by law to make certain superfund payments. With $135 billion worth of assets under management, UniSuper has grown to such a large company that when something goes wrong, the Google Cloud CEO is the one to call instead of customer service.

a June 2023 In a press release touting UniSuper’s large-scale cloud migration to Google, Sam Cooper, head of architecture at UniSuper, said: “Google Cloud VMware Engine streamlines our migration to the cloud and makes it super easy. It’s all about efficiency to provide competitive rates to our members.”

The large number of parties involved in a service means that restoring the service is not just about restoring a backup, it also involves processing all the requests and payments that had to occur during the two weeks of downtime. It meant that.