Outages happen, and they happen everywhere. Whether you leverage a public cloud, a hosting provider, or your own data center, infrastructure downtime is inevitable. Any single piece of equipment can fail, and sometimes cascading events can make an entire data center unavailable. According to a report from the Uptime Institute, on average a data center will experience one major outage and three partial outages each year.

When outages do occur, the resulting business impact can be significant. The key take away is the need to architect your applications to stay up even when your cloud or data center isn’t.

As some of you may already know, cloud 2015 is off to a stuttering start. Here is a review of some of the more interesting cloud closures of the last year.

Verizon Maintenance January 2015: Verizon’s planned maintenance for its Enterprise Cloud was completed recently, the total downtime was 40 hours. The Verizon press release that announced restoration of services states “adding seamless upgrade functionality as well as other customer-facing updates”. Let’s hope they are accurate on that. While this iteration of cloud is fairly new for Verizon (launched last fall), there is little tolerance in the crowded cloud space for downtime.

Dopbox File Sharing Fail, January 2014: A scripting glitch caused OS upgrades to be applied to active machines during a routine maintenance.  Engineers were forced to restore from backup which took 2 days due to the massive size of Dropbox’s databases.

Basecamp Closes Tent, March 2014: Project management service Basecamp suffered a DDoS attack that took it offline for 2 hours.  The attackers demanded money, but Basecamp refused to give in. The attack was a combination of SYN flood, DNS reflection, ICMP flooding, and NTP amplification with a combined flow in excess of 20Gbps.

Adobe’s Creativity Goes Down, May 2014: Over a million paying users of Adobe’s Creative Cloud and some secondary services were offline for 28 hours due to an issue an issue during database maintenance activity.

Evernote Uptime not Forever, June 2014: A DDoS attack took news aggregator Feedly and online note service, Evernote, offline for 10 hours. The perpetrators demanded money but neither service complied.

Xen Bug Causes AWS Reboots, September 2014: A previously unreported bug in the Xen hypervisor caused three days of rolling reboots to apply patches to 10% of AWS’s servers. The rest of the Xen-based cloud world performed similar updates including Rackspace and IBM/Softlayer. RightScale has a good write-up on the activity and its effects.

Infinite Loop Update Brings Down Azure, November 2014: Information Week indicates that Microsoft rolled out an update to its Azure storage service that contained an unintended infinite loop for a certain operation buried in the code which caused the service to freeze. Azure customers and a number of Microsoft properties; Office 365, Xbox Live, MSN were affected to varying degree. Most services were degraded for 4-12 hours.

Gamers Get Coal for Christmas, December 2014:  And, of course, the most recent outages to the PlayStation Network and Xbox Live, brought on by hackers, caused a great deal of angst Christmas morning.

 

What to expect in 2015?
More outages. 2014 showed that even the largest providers are susceptible to poor planning, poor engineering, unknown bugs, and malicious miscreants. While the technical issues can be prevented, we can expect to see an increase in hacker caused outages as the tools to initiate these cyberattacks become widespread and they become a preferred method of social protest and nation state domination.