Data Center Outage Puts Spotlight On Power Systems

from the the-other-grid dept

This week's outage of a major San Francisco data center has prompted a lot of discussion about the tech industry's massive energy requirements, and whether or not the existing energy infrastructure will continue to prove satisfactory. Although we blamed excessive hubris for the crash, some are pointing fingers at the utility PG&E, for letting the underlying power outage happen. Of course, this doesn't explain why 365 Main's extensive energy backup system failed to kick in as it was supposed to. Either way, it's likely that continued investments into energy systems are in order. IBM has been investing in technology that will reduce the energy demands of data centers, but the trend is helping low-tech firms as well: yesterday, engine maker Cummins reported strong earnings, due in part to the sale of generators to data centers. All of this is further evidence that tech firms are increasingly forced to get down and dirty with tangible, physical goods in order to stay competitive.

Reader Comments (rss)

(Flattened / Threaded)

  1. identicon
    SPR, Jul 27th, 2007 @ 9:39am

    Emergency Power Backup Systems

    This is ALWAYS included in any significant IT structure requiring ready access for critical data. Apparently they did not believe their data was critical.

    reply to this | link to this | view in thread ]

  2. identicon
    Bob, Jul 27th, 2007 @ 9:44am

    Re: Emergency Power Backup Systems

    Didn't you read the article? They HAD an energy backup system. It just failed to "kick in".

    reply to this | link to this | view in thread ]

  3. identicon
    Overcast, Jul 27th, 2007 @ 9:46am

    Funny now how people 'expect' machines to work right all the time...

    It's an expectation that's only result will be disappointment.

    reply to this | link to this | view in thread ]

  4. identicon
    Bigmunny, Jul 27th, 2007 @ 9:57am


    Regular testing of the system can help prevent the failed to "kick in" issue.

    reply to this | link to this | view in thread ]

  5. identicon
    Enrico Suarve, Jul 27th, 2007 @ 10:00am

    Not just the data centers

    I don't remember who posted it but in the previous thread one commentor was spot on with their analysis, that generators can spring into action all they want, but if you have an area wide power outage all they are going to do is reduce risks of data corruption from cold booting

    Basically if the entire grid goes down (as it did in this case) your connection to the data center is still only as good as the power to the first public network component not connected to your generators (i.e. still dead in the water)

    You're just a very expensive diesel powered data island

    A truly resilient solution would have to involve two completely seperate grids (and similarly seperated backup lines), or better - two very seperate locations

    reply to this | link to this | view in thread ]

  6. identicon
    Unknowledgeable Geek, Jul 27th, 2007 @ 10:23am

    Data Centers

    A lot of data centers I deal with are on two seperate grids. They have multiple internet lines coming in. The have multiple backup generators. I don't know what crappy center this was in San Fran, but I bet it was the cheapest option.

    reply to this | link to this | view in thread ]

  7. identicon
    Wes, Jul 27th, 2007 @ 10:42am


    Backup generation with UPS can give you 100% uptime. Our company has a contract with an large tech firm for such a system. They do complex benchmarks that require constant power for over 6 months at a time. The systems exist, in this case it seems like there was a catastrophic failure at either the trasfer switches or the load study was incorrectly done.

    Businesses have a tendency to add components to a electrical system without considering the output of the electrical generators and the extra load.

    reply to this | link to this | view in thread ]

  8. identicon
    SPR, Jul 27th, 2007 @ 11:32am

    Re: Re: Emergency Power Backup Systems

    It's called "TESTING"!!

    reply to this | link to this | view in thread ]

  9. identicon
    Anonymous Coward, Jul 27th, 2007 @ 11:45am

    Data Center Outage

    A good data center also stores it's data in more than one place.

    reply to this | link to this | view in thread ]

  10. identicon
    Anonymous Coward Too, Jul 27th, 2007 @ 11:57am

    Data Center Outage and Redundant Power

    My DC has redundant feeds from the street, redundant pipes, racks and racks of backup batteries. Didn't do a bit of good when a huge spike came in off the street and *vaporized* the emergency switching gear. I'm not kidding.

    No batteries, no redundant grid, no amount of testing will guarantee a no-impact failover. I've seen multiple outages at multiple sites over 20 years. The answer is IT DEPENDS. With hugs power feeds it gets complicated fast.

    You can take cheap shots if you want, doesn't mean you know anything.

    reply to this | link to this | view in thread ]

  11. identicon
    death breath, Jul 27th, 2007 @ 12:44pm

    Re: Data Center Outage

    no they don't the keep it on one floppy disk

    reply to this | link to this | view in thread ]

  12. identicon
    Chris Maresca, Jul 27th, 2007 @ 12:50pm

    According to some insiders....

    ... the problem was not that backup failed, but that they are using 90% of the incoming PGE capacity, which is far more than the generators could sustain, something like 15% more.

    The end result was a complete power meltdown.


    reply to this | link to this | view in thread ]

  13. identicon
    500CID, Jul 27th, 2007 @ 1:01pm

    Outage cause

    Outage was due to failed Basler protection relay(s)

    reply to this | link to this | view in thread ]

  14. identicon
    inc, Jul 27th, 2007 @ 1:11pm

    Re: Testing

    This is true. The data center I work in regularly tests the power conditioning, batteries and generators to ensure they are going to work when we need them. It's important that the batteries have capacity for all the critical systems, including air conditioning, and that the generators come online automatically. Also an important factor is to have contract for guaranteed fuel for that generator. When Hurricane Wilma knocked out power for 2 weeks in our DC that generator was the employee of the month.

    reply to this | link to this | view in thread ]

  15. identicon
    sean, Jul 28th, 2007 @ 1:52pm

    This is entirely their own stupid fault. Anyone who depends entirely on telecoms for their survival had better be operating out of two or more physically diverse locations. UPS and gennies are just going to keep your servers running (although I understand that even this failed, showing a remarkably cavalier attitude on behalf of Technorati, Craiglists et. al. toward ensuring that your BCP works) - to keep your telecoms up you need to be on different telecoms networks AND different power networks, just altogether in a different city, state or country.
    I wonder if I can get a hugely overpaid consulting job advising them on the basics of continuity planning?

    reply to this | link to this | view in thread ]

  16. identicon
    John W. Peace II CEcD, Aug 5th, 2007 @ 8:24am

    Safest Place on Earth for Data Centers!!!

    Natural disasters, power outages, terrorism, not a problem if you locate your mission critical operations in the Safest Place on Earth!

    Take a look at:

    reply to this | link to this | view in thread ]

  17. Re: Data Center Outage and Redundant Power

    An appropriate sized REGULATOR(s) before the panels and even UPS's are meant to absorb and take out all power spikes from city power. Sometimes, we forget to add them to our systems. Their only purpose is to prevent power spikes from continuing on to your infrastructure...

    reply to this | link to this | view in thread ]

  18. identicon
    John Peace, Jul 3rd, 2009 @ 3:54pm

    Safest Place on Earth for Data Centers!

    Looking for the safest place to locate Data Centers?

    Locate @ the power source with little risk and cheaper electrical rates!

    reply to this | link to this | view in thread ]

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
Insider Shop - Show Your Support!

Hide this ad »
Essential Reading
Techdirt Deals
Hide this ad »
Techdirt Insider Chat
Hide this ad »
Recent Stories
Advertisement - Amazon Prime Music
Hide this ad »


Email This

This feature is only available to registered users. Register or sign in to use it.