Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?
from the i'd-say-worse... dept
As you may have heard, yesterday web host/domain registrar GoDaddy had some serious downtime, taking tons of sites down. Much of the news coverage focused on a single hacker who tried to take credit for bringing them down. However, now that GoDaddy is back, it insists the problem was entirely internal and not the result of a hack. The company was quite explicit on that point:
The service outage was not caused by external influences. It was not a “hack” and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented measures to prevent this from occurring again.
At no time was any customer data at risk or were any of our systems compromised.
While my first reaction to all of this was to wonder who would still use GoDaddy, my second question is to wonder whether GoDaddy looks better or worse if it was its own fault that the service went down so broadly. Mistakes happen, but a company like GoDaddy survives on its ability not to make mistakes at that level. I guess, in the end, it’s just yet another reminder of why people might want to look for alternatives.
Filed Under: ddos, downtime, hacking, internal problems
Companies: godaddy
Comments on “Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?”
One Here
My employer’s website uses GoDaddy. It’s probably not a big deal, though, since almost nobody visits it.
Re: One Here
Is your employer a Nigerian scam artist? Where do I apply?
One tried to scam me this morning.
Funny thing, apparently sites that aren’t even hosted by GoDaddy also went down.
If the domain name was OWNED by GoDaddy, purchased and hosted elsewhere, the site still went down and displayed a GoDaddy error message.
Re: Re:
If that’s the case then I would guess that the DNS tables at GoDaddy were corrupted, or possibly the routing tables to the DNS servers. If you can get to GoDaddy’s DNS servers, but they can’t resolve a name, that’s when you get the GoDaddy error page.
Re: Re: Re:
GoDaddy had to be hosting the DNS for the domain, not just be the registrar. I don’t do any web hosting with them, but I have a couple of personal domains that they run the DNS for and they went down. My company’s DNS is registered there but we run our own DNS servers and those domains were unaffected.
I like how the outage started with “a series of internal network events”, and then was fixed when “we took corrective actions”.
Typical (lame) network guys not taking responsibility for causing a problem (or creating the situation for the problem to occur), and then swooping in as the hero.
Re: Re:
You haven’t seen some of the crazy shit that Sysadmins need to do sometimes to get things to work.
Re: Re:
Sounds like a classical BGP routing table corruption. Happens all the time, everywhere. Something like three times a year at a typical ISP, but usually they get fixed within minutes to a few hours. But sometimes, every few years, something bigger happens, which takes hours to days to fix, and that’s probably what happend there.
After their support for SOPA my compassion meter is barely registering, and my “give a fuck” has apparently left the building. Honestly, the lulzyest part is that the media still thinks random twitter accounts with variations of the word “anonymous” in them, count as a reliable source.
Re: Re:
Yeah, I didn’t buy the anon angle at all.
They don’t seem to the types that would do haxxorz that don’t have a political protest motivation.
Forgive me for I have sinned...
I am still on GoDaddy… Not because I like them, but because I didn’t cancel my webhosting before they auto-renewed my subscription… AND, I am also a tad lazy and don’t feel like navigating their vomit style account pages.
BTW, Is Netflix down now? I can’t resolve it… Sadness 🙁
Re: Forgive me for I have sinned...
Netflix streaming isn’t down 🙂
This was actually just being discussed on the NANOG mailing list.
If you admit that a hack took place, you now have to investigate whether any information was also taken during the hack, which is costly and a lengthy process. This, in fact, will continue bad press for a long time. By giving a we screwed up routing excuse, they can then dismiss accountability until a investigation has been conducted. This gives them time to research to give a valid RFO, and delay any bad press in the meantime. It pretty much makes sense from both a legal standpoint, and a PR standpoint imho.
Being a network guy, I understand that these problems can arise, but the length of the outage tends to cast doubts on what exactly was going on. If this was a major network overhaul gone completely bad, I can understand. It can take quite some time to rollback a major overhaul across multiple sites, but not enough information was given to sway me one way or another.
Choices...
We were stupid and got hacked.
vs
We are just inept at running our systems.
There is no real good way out of this, the upside is they can claim they weren’t hacked so they look like the good guys… except they screwed their customers long and hard all on their own then…
Another reason to not use GoDaddy.
Re: Choices...
At least they had the balls to admit they screwed up, that’s something. They could have said they got hacked, but their security kicked in and shut everything down before data could get stolen. That way they wouldn’t have to do any actual investigation and they look like they have advanced security in place.
Re: Re: Choices...
They still don’t have to do any actual investigation.
We have a lone hacker taking credit vs GoDaddy, I do not like my odds with either side.
It reminds me of the Sony hacks, where Sony kept claiming they were never breached. They were doing stuff on their own, and only after the evidence reached stupid proportions did they finally admit they were hacked. Then they “found” a single text file with a well known motto to shift the blame to a group that never took credit for the hack. Then it came out how some of the hacks happened was because they skipped over even basic security concepts, and ignored people pointing out glaring flaws in the system.
Hacks in general happen, we all know this, but it seems to admit to them is worse than being hacked.
The only secure computer is one not connected to the net and not allowed human interaction.
If they came out and admitted we got hacked, and we have fixed the problem most people wouldn’t think any less of them. It would be a good reminder to stay up to date with your security, and never just assume your secure.
Instead we get PR speak trying to save the company’s reputation with doublespeak and hedging around the issues. Companies want to treat getting hacked as a corporate secret that no one can ever know about, and all that does it make us all less secure.
Re: Re: Re: Choices...
Is there any possibly better way to approach this, though, other than (arguably) admitting a (possible) screw-up and taking flak still?
Re: Re: Re:2 Choices...
You add more to the mix, like the simple fact they are a massive registrar.
And then someone over at ars mention in the comments of the story that a DDOS started the problem, then it all went sideways, and they are playing buzzword bingo to make it look like a little booboo from inside rather than the outside.
I take it with a grain a salt, because on the internet no one knows your a dog… but lots of people are suggesting this doesn’t add up as explained unless they were not following even basic rules. Sort of like that cert company who claimed long and hard they weren’t hacked, even as people were pulling up bogus certs generated from their systems.
GoDaddy? How about NoDaddy? One would have to have a double lobotomy to willingly use their service these days.
It was not a hack…because GoDaddy said so. And that’s the bottom line!
Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it’s an incredible hard job to isolate the source and to reset each part of the network.
GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.
I know you want to kick them because you hate them and all, but come on Mike, can’t you be reasonable and accept that shit happens sometimes?
Re: Re:
Please name the names, times, and dates of the last 5 hosts who went dark because of a cascade router failure.
Mike did not kick them, Mike pointed out a news story involving *gasp* tech and GoDaddy’s statement that says it was not a hack.
Mike wondered how a mistake like this can happen in a company that is so big, to be kicking them he would have written a piece being much more critical of them and pointing out the massive amount of fail and underhanded tactics they employee. This was merely commentary on a single event and the “response” by GoDaddy.
Re: Re: Re:
Probably because we expect a lot in spite of being human. It happens, heh.
Re: Re:
“Cascade router failures can happen to almost anyone”
Plan B? – What’s that?
“Quite simply, it happens”
Interesting that it happens so little elsewhere, maybe other networks are designed for fault tolerance. Naaa – that can’t be it.
“I know you want to kick them because you hate them and all”
Awww, isn’t that special.
Re: Re:
Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it’s an incredible hard job to isolate the source and to reset each part of the network.
GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.
I know you want to kick them because you hate them and all, but come on Mike, can’t you be reasonable and accept that shit happens sometimes?
Please stop disrupting the Techdirt narrative. Any company that supported SOPA is evil, incompetent, should be boycotted and their services suck. Masnick is in a blind rage because after all of his self-congratulations about how he somehow had a stake in defeating SOPA, the end result was a big zero was all of the SOPA players side stepping and using private agreements to accomplish much the same thing.
Re: Re: Re:
You’re obsessed with pushing this point on, aren’t you?
Would publicising, legalising, and setting in stone private agreements to leverage further harsher and narrow laws have been an improvement?
Re: Re: Re:
Are you as dumb as you sound?
Re: Re:
I actually agree with this poster. It feels like TechDirt badly wanted to make hay out of this news, and the take of “Which is worse?” feels ham-handed.
PS, I hate GoDaddy too!
Re: Re:
“Cascade router failures can happen to almost anyone.”
“GoDaddy is not immune to the problems of having a sizable network.”
That’s the thing, they shouldn’t have just one network. When you’re making something at this scale one of the design considerations is fault tolerance. The way to achieve that is by designing the system to be separate parts that do not rely on each other. No single change should EVER be able to affect all of their DNS infrastructure all at once like this, it should always require changing it in at least 2 places (for a global DNS infrastructure I’d require at least 6: one per continent, and we’ll risk all the customers in Antarctica by lumping them in with someone else)
Re: Re: Re:
Please stop disrupting the troll narrative. Any post that criticized SOPA supporting corporations is evil, wrong and should be berated. SOPA supporting trolls are in a blind rage because after all of their self-congratulations and gloating about how they will eventually rule the world, the end result was a big zero as all of the SOPA players got bitch slapped and decided to side step the law and use private agreements to accomplish much the same thing.
I know, it’s because SOPA didn’t pass. Er…
Man I’ve heard some bull, but this one is really good. Huge operation like GoDaddy and no backup routers. Looks like a lot of people are going to get fired today!
GoDaddy’s been hacked for years. There are all kind of back doors that let not reputable people put directories on peoples sites. I know from experience. One day I was editing my site and found a directory with an web page containing an login form for a bank. I didn’t put it there and when I told GoDaddy they said it was my fault. I needed to change my password more. Out of there, haven’t been hacked since.
I believe you Godaddy. Just as much as I believe that you are genuinely against SOPA and not just for PR reasons from the boycott that didn’t change shit since apparently people are still using Godaddy.
Re:
BGP errors are always cascading due to the nature of the protocol, so this is rather easy to show 5 times or more that this has happened on a global scale.
1. BGP leak on Dery Telecom Inc in Ontario
2. BGP leak on Telstra in Australia
3. BGP leak on SK Telecom in California
4. BGP leak on China Telecom in China
5. BGP leak on Evolva Telecom in Romania
Re:
I think you posted under the wrong comment.
In answer to Brad C, I can only say: Royal Bank of Scotland (RBS).
MSM Spin
I personally really don’t care about GoDaddy, but the really intreasting thing in all this I saw was the brief mention on Good Morning America (ABC/Disney).
Josh Elliot mentioned the story while doing the news yesterday morning. I can’t find a link.
Basically he said that GoDaddy had been taken down reportedly by the internet group Anonymous because they were angry about them supporting laws that could stop them from pirating movies and stuff.
Woohoo Disney! Way to spin a news report, even when you don’t even have ANY of the facts!
Oof. I might have misspoke. I found the video for GMA’s yesterday episode and couldn’t find the mention. I think I may have mixed up coverage from my local ABC affiliate with the GMA coverage and the local newscaster was the one who stated it like that. Of course the local affiliate doesn’t put the full newscasts online, so I won’t ever find it now.
Re:
Ummm no, just changing the words found in a prior post for the lulz
Offical RFO from GoDaddy
GoDaddy
Basically, eBGP redistribution into iGP crashed the routers due to a bug in the OS. Due to customer traffic and routing traffic continually overloading the routers, they had to limit the traffic and delay recovery over a period of time causing the extended outage.
It’s a pretty detailed RFO for end-users, and I give them credit for supplying such details.
downtime
My website and business lives and dies by being available to my customers. DNS Services dnssvc.com offers a product that’s so efficient at making sure my website doesn?t go down with a price to match. I will never leave my website unprotected again.
Are you as dumb as you sound?