Two terrifying tales of the Dark Cloud:
Something bad happened yesterday. A cooler broke down in the hosting center that we run the Chat service in. This event lead in to two catastrophic consequences from our point of view. Firstly we had to ramp down the service down for a very long period, in fact most of the yesterday morning and afternoon we had a service break. Secondly, our database broke down. Despite the fact that we had regular back-ups, we were not able to set it right.
What we had to eventually do, was to return a back-up copy from our previous hosting center, created on the 23rd January 2009. As a consequence, anything that you’ve done since that (profile details, images, friendships) are gone. The problem has now been fixed and the server is now back and working. You should now be able to log back to your Contacts and start using it as normal. You do, however, need to invite your friends again etc.
Imagine that! Imagine meeting someone new, entering their Contact data, syncing to the Cloud — and then having a crash on your own phone, requiring restoration from the Cloud. And then you’re informed — Poof! — “Your data. We eated it!”
As if that wasn’t bad enough, what if an entire service you’ve set up in the Cloud goes FAIL?
Here’s a horrifying tale of catastrophe from Gear Diary: Surviving the Nuclear Winter after an Atomic Server Crash, Part One
About 9:30 I finally crawled out of the world’s most comfortable bed, had some cereal, played with Avah, watched CNN for a bit, and then sat in front of my computer to take a look at the morning’s email. I noticed that there were a bunch of comments pending moderation, so instead of doing them individually through the email interface, I logged into our Admin panel. I approved the first comment, and got a database error. Anyone who has ever used WordPress has seen errors before, and many times a quick refresh will right them. I backed up and tried approving comments again, and got the same error.
As I always do when something back-end happens that I can’t figure out, I called Ewdi – website code guru extraordinaire – catching him at lunch. He said he would get online as soon as he returned home. So in the meantime, I sat here waiting…hoping against hope that the site would come back online. Hoping that when Ewdi came on I could laughingly tell him “never mind, and I’m truly sorry for disturbing your lunch!”
But instead, I got to go through one of the most frustrating, depressing, and mind-numbing events any web-master could possibly endure.
It’s truly a gruesome tale that is worse than anything Stephen King could imagine (although now that he has a Kindle, maybe not!). It’s long but worth the read for an insight about what can happen behind the scenes with the technology we might have to rely on.
It’s going to be very interesting to see what strategies evolve as people attempt to maintain data-safe lives.
How many backups will be enough?