Blame Murphy's Law, Excessive Hubris For Blackouts In SF

from the or-blame-the-PR-people,-your-choice dept

Lots of San Francisco-based web sites (Craigslist, Six Apart, Yelp, and Technorati, among others) have been experiencing some problems today, after power outages in the city took down 365 Main, a major hosting facility there. While the power company says it doesn’t yet know what’s caused the outages, we have a pretty good idea: a 365 Main press release that went out this morning, bragging about the two years of continuous uptime one of its customers has had since moving to the data center. And when these guys invoke Murphy’s Law, they don’t do it by half, either. The release also brags about the center’s “unique billing system in which 365 Main only charges customers for the exact amount of power that is used” — so presumably today will be free. But what really sealed their fate was this paragraph:

“To ensure uptime for key tenants such as RedEnvelope, 365 Main provides modern power and cooling infrastructure. The company’s San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.”

Good to see those backup systems are working!

Filed Under:
Companies: 365 main, redenvelope

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Blame Murphy's Law, Excessive Hubris For Blackouts In SF”

Subscribe: RSS Leave a comment
dorpus says:


I once worked for Motorola’s Iridium project, which had spent millions of dollars on redundant power backup systems at their controlling facility. However, it was still not immune to the stray rat who bit into a power cable — that would have taken the facility down for days. The facility was near the Potomac River, so there were lots of rats.

Russia’s Baikonur Kosmodrome also had a serious problem with rats chewing cables at the time, and they dealt with the problem by keeping cats.

Enrico Suarve says:

Re: well duh!

You power them off UPS and use this to manage the switchover (humans just aren’t fast or reliable enough)

Something clearly went wrong however. I had a client once who had a similar failure, the batteries switched over to the generator which started just fine and all was OK, but a few minutes later the generator stopped

Cause: whichever plank had installed the generator had wired up its electric diesel pump to the mains only, instead of to the generators own electrical output…. it drank itself dry ;0)

Anonymous Coward says:

uhh.. they probably run huge diesel gensets that are capable of putting out Megawatts of power. There is no need for offsite mirrors. The diesels can be at peak load in under 1 minute. UPS can hold you over for the time in between. Also, most facilities have automagical transfer switches. Proper maintenance of the gensets (that means running them under loadbanks, and having qualified people come in to do the oil and other fluids) will almost guarantee that your facility will be online in minutes rather than hours.

Anonymous Coward Also says:

you think you're so smart...

My DC has redundant feeds from the street, redundant pipes, racks and racks of backup batteries. Didn’t do a bit of good when a huge spike came in off the street and *vaporized* the emergency switching gear. I’m not kidding.

No batteries, no redundant grid, no amount of testing will guarantee a no-impact failover. I’ve seen multiple outages at multiple sites over 20 years. The answer is IT DEPENDS. With hugs power feeds it gets complicated fast.

It is incredibly ironic that the PR folks put out a release like that the same day… but really, most hosting sites say the same kinds of things. You can take cheap shots if you want, doesn’t mean you know anything. I often enjoy the hubris of the media!

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...