Amazon Cloud Outage – The lessons

Friday, 22 April 2011

Over the past couple of years, well meaning people in the cloud industry have told me “You ought to host on a PaaS (Platform-as-a-Service) like Amazon or Google. Your customers would be reassured by having such a big name behind you, and it solves all the scalability issues”. And I’ve replied that, call us old fashioned, but we like to know where our customers’ data is and we like to have control of the technical environment we’re running on, and you only get both if you own and maintain your own servers. There is also the mostly-completey-ignored issue of complying with UK & European law and not holding data on EC citizens outside of the EC.

This blog post is not about schadenfreude, rejoicing in Amazon EC2’s two day outage that has taken a swathe of major cloud applications down, including some of our competitors. This is a plea (yet again!) for simplicity in IT design.

It is a truism that the more complex a system, the greater the chance that something will go wrong. The more firewalls, load balancers, routers and software layers between the customer’s browser and your application, the greater chance that something will fail, be it as simple as an engineer in the datacentre pulling out the wrong cable (as happened to us a few months ago).

The other reason we like hosting our own servers is that, if they go down, we have a team of our own people working flat out focussed 100% on getting our system back and not 1,000 other systems at the same time. Which is a lot easier job, especially as we’ve made sure that we have as few layers between our boxes and the outside world.

We also have a backup system on standby with real time data sync so that if our main datacentre does go down, we can fail over in about 20 minutes.

So, cloud developers! Rack your own boxes and keep the IT simple. Maintaining servers is not that hard, you’ll get much better scalability and efficiency by specifying your own software and hardware platform. And your customers won’t be left without an application that they have paid for.

Just make sure there are no Armenian old ladies near the building.