This was a bit of news that almost passed under the radar unnoticed. As a matter of fact it would have gone unnoticed but for the fact that an incident report was released by Google’s cloud computing service division.
The incident was that between Thursday 13th – Monday 17th August, a number of lightening strikes caused a major power outage in one of Google’s storage centres in Europe causing permanent damage to some disks which in effect meant permanent loss of some clients data.
Most people around the world did not notice this glitch because the damage was limited to only a part of its data centre in Belgium. The data centre is located in the town of St. Ghislam, a rather small town about 50 kilometers South West of Brussels, the capital of Belgium. This is the town the Google Compute Engine (GCE) service located it’s europe-west1-b zone. This zone is part of the larger europe-west1 region. The other regions are US-central1 based in Bluffs, Iowa and asia-east1 based in Changhua County in Taiwan. All these regions are further divided into zones and it is one of the data centres in the europe-west1-b zone that got hit by the lightening strikes.
Normally a massive data centre like the GCE’s service are normally equipped with lightening protectors far more advanced than the ones found in normal homes. So how did the lightening strikes take out this massively protected centre. According to reports, lightening hit the area four times in a short period of time, which in normal everyday occurrence is a highly improbable thing to happen. But it must be down to extreme bad luck that the area was hit not twice but four times. And according to electric and electronic experts with experience in this sort of thing, even if the lightening struck the area a couple of kilometres from the data centre, the tremendous electrical shock wave can travel through cabling and cause considerable damage in the data centre by fusing overheated circuits.
The Google Compute Engine service allows clients all over the world who number in their millions to store data and do cloud computing from around the world. Google Compute Engine did not release the number of clients affected but said only 0.000001%of disk space was affected. Of course one would expect them to try and downplay the real extend of the damage caused by the lightening like any responsible corporate organization. But the fact there is almost zero outcry from clients around the world shows that they are probably telling the truth.
But another surprising thing is the fact that such huge data centres normally plan for power outages like this with the installation of state of the art redundancy systems that trigger on auxiliary power supply from on-board batteries that act as effective backup for situations like these. But some of the servers packed up because of, “extended or repeated battery drain,” according to the incident report from the cloud services Division of Google. The report went further to say, “In almost all cases, the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state.” The full report can be found here.
The cheering news is that like a responsible global corporation, Google has already deployed its infrastructure engineers and they are currently replacing the storage hardware with robust and resilient ones that can withstand power outages of such magnitude.
The cloud computing service division is also advising its clients to avail themselves of the other regions’ infrastructure when storing their data. This would act as cloud backup for their important data in case, God forbid, lightening strikes a fifth time.