Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?

from the i'd-say-worse... dept

As you may have heard, yesterday web host/domain registrar GoDaddy had some serious downtime, taking tons of sites down. Much of the news coverage focused on a single hacker who tried to take credit for bringing them down. However, now that GoDaddy is back, it insists the problem was entirely internal and not the result of a hack. The company was quite explicit on that point:

The service outage was not caused by external influences. It was not a “hack” and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented measures to prevent this from occurring again.

At no time was any customer data at risk or were any of our systems compromised.

While my first reaction to all of this was to wonder who would still use GoDaddy, my second question is to wonder whether GoDaddy looks better or worse if it was its own fault that the service went down so broadly. Mistakes happen, but a company like GoDaddy survives on its ability not to make mistakes at that level. I guess, in the end, it’s just yet another reminder of why people might want to look for alternatives.

Filed Under: , , ,
Companies: godaddy

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?”

Subscribe: RSS Leave a comment
42 Comments
Anonymous Coward says:

This was actually just being discussed on the NANOG mailing list.

If you admit that a hack took place, you now have to investigate whether any information was also taken during the hack, which is costly and a lengthy process. This, in fact, will continue bad press for a long time. By giving a we screwed up routing excuse, they can then dismiss accountability until a investigation has been conducted. This gives them time to research to give a valid RFO, and delay any bad press in the meantime. It pretty much makes sense from both a legal standpoint, and a PR standpoint imho.

Being a network guy, I understand that these problems can arise, but the length of the outage tends to cast doubts on what exactly was going on. If this was a major network overhaul gone completely bad, I can understand. It can take quite some time to rollback a major overhaul across multiple sites, but not enough information was given to sway me one way or another.

That Anonymous Coward (profile) says:

Choices...

We were stupid and got hacked.
vs
We are just inept at running our systems.

There is no real good way out of this, the upside is they can claim they weren’t hacked so they look like the good guys… except they screwed their customers long and hard all on their own then…

Another reason to not use GoDaddy.

Chronno S. Trigger (profile) says:

Re: Choices...

At least they had the balls to admit they screwed up, that’s something. They could have said they got hacked, but their security kicked in and shut everything down before data could get stolen. That way they wouldn’t have to do any actual investigation and they look like they have advanced security in place.

That Anonymous Coward (profile) says:

Re: Re: Choices...

They still don’t have to do any actual investigation.
We have a lone hacker taking credit vs GoDaddy, I do not like my odds with either side.

It reminds me of the Sony hacks, where Sony kept claiming they were never breached. They were doing stuff on their own, and only after the evidence reached stupid proportions did they finally admit they were hacked. Then they “found” a single text file with a well known motto to shift the blame to a group that never took credit for the hack. Then it came out how some of the hacks happened was because they skipped over even basic security concepts, and ignored people pointing out glaring flaws in the system.

Hacks in general happen, we all know this, but it seems to admit to them is worse than being hacked.
The only secure computer is one not connected to the net and not allowed human interaction.
If they came out and admitted we got hacked, and we have fixed the problem most people wouldn’t think any less of them. It would be a good reminder to stay up to date with your security, and never just assume your secure.
Instead we get PR speak trying to save the company’s reputation with doublespeak and hedging around the issues. Companies want to treat getting hacked as a corporate secret that no one can ever know about, and all that does it make us all less secure.

That Anonymous Coward (profile) says:

Re: Re: Re:2 Choices...

You add more to the mix, like the simple fact they are a massive registrar.

And then someone over at ars mention in the comments of the story that a DDOS started the problem, then it all went sideways, and they are playing buzzword bingo to make it look like a little booboo from inside rather than the outside.

I take it with a grain a salt, because on the internet no one knows your a dog… but lots of people are suggesting this doesn’t add up as explained unless they were not following even basic rules. Sort of like that cert company who claimed long and hard they weren’t hacked, even as people were pulling up bogus certs generated from their systems.

Anonymous Coward says:

Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it’s an incredible hard job to isolate the source and to reset each part of the network.

GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.

I know you want to kick them because you hate them and all, but come on Mike, can’t you be reasonable and accept that shit happens sometimes?

That Anonymous Coward (profile) says:

Re: Re:

Please name the names, times, and dates of the last 5 hosts who went dark because of a cascade router failure.

Mike did not kick them, Mike pointed out a news story involving *gasp* tech and GoDaddy’s statement that says it was not a hack.

Mike wondered how a mistake like this can happen in a company that is so big, to be kicking them he would have written a piece being much more critical of them and pointing out the massive amount of fail and underhanded tactics they employee. This was merely commentary on a single event and the “response” by GoDaddy.

abc gum says:

Re: Re:

“Cascade router failures can happen to almost anyone”

Plan B? – What’s that?

“Quite simply, it happens”

Interesting that it happens so little elsewhere, maybe other networks are designed for fault tolerance. Naaa – that can’t be it.

“I know you want to kick them because you hate them and all”

Awww, isn’t that special.

Anonymous Coward says:

Re: Re:

Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it’s an incredible hard job to isolate the source and to reset each part of the network.

GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.

I know you want to kick them because you hate them and all, but come on Mike, can’t you be reasonable and accept that shit happens sometimes?

Please stop disrupting the Techdirt narrative. Any company that supported SOPA is evil, incompetent, should be boycotted and their services suck. Masnick is in a blind rage because after all of his self-congratulations about how he somehow had a stake in defeating SOPA, the end result was a big zero was all of the SOPA players side stepping and using private agreements to accomplish much the same thing.

Brad C (profile) says:

Re: Re:

“Cascade router failures can happen to almost anyone.”
“GoDaddy is not immune to the problems of having a sizable network.”

That’s the thing, they shouldn’t have just one network. When you’re making something at this scale one of the design considerations is fault tolerance. The way to achieve that is by designing the system to be separate parts that do not rely on each other. No single change should EVER be able to affect all of their DNS infrastructure all at once like this, it should always require changing it in at least 2 places (for a global DNS infrastructure I’d require at least 6: one per continent, and we’ll risk all the customers in Antarctica by lumping them in with someone else)

abc gum says:

Re: Re: Re:

Please stop disrupting the troll narrative. Any post that criticized SOPA supporting corporations is evil, wrong and should be berated. SOPA supporting trolls are in a blind rage because after all of their self-congratulations and gloating about how they will eventually rule the world, the end result was a big zero as all of the SOPA players got bitch slapped and decided to side step the law and use private agreements to accomplish much the same thing.

Anonymous Coward says:

GoDaddy’s been hacked for years. There are all kind of back doors that let not reputable people put directories on peoples sites. I know from experience. One day I was editing my site and found a directory with an web page containing an login form for a bank. I didn’t put it there and when I told GoDaddy they said it was my fault. I needed to change my password more. Out of there, haven’t been hacked since.

Anonymous Coward says:

Re:

Please name the names, times, and dates of the last 5 hosts who went dark because of a cascade router failure.

BGP errors are always cascading due to the nature of the protocol, so this is rather easy to show 5 times or more that this has happened on a global scale.
1. BGP leak on Dery Telecom Inc in Ontario
2. BGP leak on Telstra in Australia
3. BGP leak on SK Telecom in California
4. BGP leak on China Telecom in China
5. BGP leak on Evolva Telecom in Romania

Gwiz (profile) says:

MSM Spin

I personally really don’t care about GoDaddy, but the really intreasting thing in all this I saw was the brief mention on Good Morning America (ABC/Disney).

Josh Elliot mentioned the story while doing the news yesterday morning. I can’t find a link.

Basically he said that GoDaddy had been taken down reportedly by the internet group Anonymous because they were angry about them supporting laws that could stop them from pirating movies and stuff.

Woohoo Disney! Way to spin a news report, even when you don’t even have ANY of the facts!

Gwiz (profile) says:

Oof. I might have misspoke. I found the video for GMA’s yesterday episode and couldn’t find the mention. I think I may have mixed up coverage from my local ABC affiliate with the GMA coverage and the local newscaster was the one who stated it like that. Of course the local affiliate doesn’t put the full newscasts online, so I won’t ever find it now.

Anonymous Coward says:

Offical RFO from GoDaddy

GoDaddy

Basically, eBGP redistribution into iGP crashed the routers due to a bug in the OS. Due to customer traffic and routing traffic continually overloading the routers, they had to limit the traffic and delay recovery over a period of time causing the extended outage.

It’s a pretty detailed RFO for end-users, and I give them credit for supplying such details.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...