Last week Amazon EC2 Cloud when down to a cascade power failure as a result of thunderstorms.
Here's Amazon's description -
==================
"At approximately 8:44PM PDT, there was a cable fault in the high voltage Utility power distribution system. Two Utility substations that feed the impacted Availability Zone went offline, causing the entire Availability Zone to fail over to generator power. All EC2 instances and EBS volumes successfully transferred to back-up generator power."
"At 8:53PM PDT, one of the generators overheated and powered off because of a defective cooling fan. At this point, the EC2 instances and EBS volumes supported by this generator failed over to their secondary back-up power (which is provided by a completely separate power distribution circuit complete with additional generator capacity).
Unfortunately, one of the breakers on this particular back-up power distribution circuit was incorrectly configured to open at too low a power threshold and opened when the load transferred to this circuit. After this circuit breaker opened at 8:57PM PDT, the affected instances and volumes were left without primary, back-up, or secondary back-up power."
==================
Although Amazon appeared to have the proper power redundancy and failover capabilities in place, they obviously didn't work. In addition, there was no data failover to a secondary site., so a sinlge point of failure in a chain and the datacenter went off the grid. Poof. All the users and consumers of the services stored in that part of the cloud were at the mercy of the Amazon engineers and support technicians to restore service. Even if the customers wanted to failover themselves, they couldn't access anything within that part of the cloud to throw the switch. Dead in the water.
People can blame Amazon till they are blue in the face. Their testing was insufficient, their infrastructure relied components to close to together, they suck. At the end of the day there was a powerful natural disaster and no company and no one can expect every What-If scenario to be ran. Time is money and nothing would ever be released if this was the practice. What this shows is that The Cloud is not the magical marketing wonderland it has been made out to be.
The Cloud is out of your control, period. You are at the mercy of a contract Service Level Agreement (SLA) and when it hits the fan you are at the mercy of the provider. You are also at the mercy of the provider and most likely have to clue or say into the level of expertise they hire to maintain The Cloud. Luckily Amazon is the trend setter on Cloud customer service, communication and response. Other Cloud services, not so much. But this is The Cloud reality and what you are getting into when you take the plunge to the sky.
Like with all technology get educated before you allow the Brochure Buzzwords and Conference Presentations decide your path for you. There are risks in everything, it's up to you to decide how much you are willing to take.
End of Line.
Although Amazon appeared to have the proper power redundancy and failover capabilities in place, they obviously didn't work. In addition, there was no data failover to a secondary site., so a sinlge point of failure in a chain and the datacenter went off the grid. Poof. All the users and consumers of the services stored in that part of the cloud were at the mercy of the Amazon engineers and support technicians to restore service. Even if the customers wanted to failover themselves, they couldn't access anything within that part of the cloud to throw the switch. Dead in the water.
People can blame Amazon till they are blue in the face. Their testing was insufficient, their infrastructure relied components to close to together, they suck. At the end of the day there was a powerful natural disaster and no company and no one can expect every What-If scenario to be ran. Time is money and nothing would ever be released if this was the practice. What this shows is that The Cloud is not the magical marketing wonderland it has been made out to be.
The Cloud is out of your control, period. You are at the mercy of a contract Service Level Agreement (SLA) and when it hits the fan you are at the mercy of the provider. You are also at the mercy of the provider and most likely have to clue or say into the level of expertise they hire to maintain The Cloud. Luckily Amazon is the trend setter on Cloud customer service, communication and response. Other Cloud services, not so much. But this is The Cloud reality and what you are getting into when you take the plunge to the sky.
Like with all technology get educated before you allow the Brochure Buzzwords and Conference Presentations decide your path for you. There are risks in everything, it's up to you to decide how much you are willing to take.
End of Line.
Posted in: BrochureBuzzwords,News,Technology
2 comments:
Dude, is that your picture? If so, awesome.. :)
I wish it was, it's a famous stock photo, not photoshopped.
Post a Comment