Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?

from the i'd-say-worse... dept

As you may have heard, yesterday web host/domain registrar GoDaddy had some serious downtime, taking tons of sites down. Much of the news coverage focused on a single hacker who tried to take credit for bringing them down. However, now that GoDaddy is back, it insists the problem was entirely internal and not the result of a hack. The company was quite explicit on that point:
The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented measures to prevent this from occurring again.

At no time was any customer data at risk or were any of our systems compromised.
While my first reaction to all of this was to wonder who would still use GoDaddy, my second question is to wonder whether GoDaddy looks better or worse if it was its own fault that the service went down so broadly. Mistakes happen, but a company like GoDaddy survives on its ability not to make mistakes at that level. I guess, in the end, it's just yet another reminder of why people might want to look for alternatives.


Reader Comments (rss)

(Flattened / Threaded)

  •  
    icon
    A Dan (profile), Sep 11th, 2012 @ 4:46pm

    One Here

    My employer's website uses GoDaddy. It's probably not a big deal, though, since almost nobody visits it.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 11th, 2012 @ 4:48pm

    Funny thing, apparently sites that aren't even hosted by GoDaddy also went down.

    If the domain name was OWNED by GoDaddy, purchased and hosted elsewhere, the site still went down and displayed a GoDaddy error message.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      Chronno S. Trigger (profile), Sep 11th, 2012 @ 5:13pm

      Re:

      If that's the case then I would guess that the DNS tables at GoDaddy were corrupted, or possibly the routing tables to the DNS servers. If you can get to GoDaddy's DNS servers, but they can't resolve a name, that's when you get the GoDaddy error page.

       

      reply to this | link to this | view in chronology ]

      •  
        icon
        Brad C (profile), Sep 11th, 2012 @ 10:41pm

        Re: Re:

        GoDaddy had to be hosting the DNS for the domain, not just be the registrar. I don't do any web hosting with them, but I have a couple of personal domains that they run the DNS for and they went down. My company's DNS is registered there but we run our own DNS servers and those domains were unaffected.

         

        reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 11th, 2012 @ 4:50pm

    I like how the outage started with "a series of internal network events", and then was fixed when "we took corrective actions".

    Typical (lame) network guys not taking responsibility for causing a problem (or creating the situation for the problem to occur), and then swooping in as the hero.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      The eejit (profile), Sep 12th, 2012 @ 1:08am

      Re:

      You haven't seen some of the crazy shit that Sysadmins need to do sometimes to get things to work.

       

      reply to this | link to this | view in chronology ]

    •  
      icon
      Seegras (profile), Sep 12th, 2012 @ 3:12am

      Re:

      Sounds like a classical BGP routing table corruption. Happens all the time, everywhere. Something like three times a year at a typical ISP, but usually they get fixed within minutes to a few hours. But sometimes, every few years, something bigger happens, which takes hours to days to fix, and that's probably what happend there.

       

      reply to this | link to this | view in chronology ]

  •  
    icon
    Anonymous Coward (profile), Sep 11th, 2012 @ 4:52pm

    After their support for SOPA my compassion meter is barely registering, and my "give a fuck" has apparently left the building. Honestly, the lulzyest part is that the media still thinks random twitter accounts with variations of the word "anonymous" in them, count as a reliable source.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, Sep 11th, 2012 @ 7:30pm

      Re:

      Yeah, I didn't buy the anon angle at all.

      They don't seem to the types that would do haxxorz that don't have a political protest motivation.

       

      reply to this | link to this | view in chronology ]

  •  
    icon
    robofog (profile), Sep 11th, 2012 @ 4:55pm

    Forgive me for I have sinned...

    I am still on GoDaddy... Not because I like them, but because I didn't cancel my webhosting before they auto-renewed my subscription... AND, I am also a tad lazy and don't feel like navigating their vomit style account pages.

    BTW, Is Netflix down now? I can't resolve it... Sadness :(

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 11th, 2012 @ 5:03pm

    This was actually just being discussed on the NANOG mailing list.

    If you admit that a hack took place, you now have to investigate whether any information was also taken during the hack, which is costly and a lengthy process. This, in fact, will continue bad press for a long time. By giving a we screwed up routing excuse, they can then dismiss accountability until a investigation has been conducted. This gives them time to research to give a valid RFO, and delay any bad press in the meantime. It pretty much makes sense from both a legal standpoint, and a PR standpoint imho.

    Being a network guy, I understand that these problems can arise, but the length of the outage tends to cast doubts on what exactly was going on. If this was a major network overhaul gone completely bad, I can understand. It can take quite some time to rollback a major overhaul across multiple sites, but not enough information was given to sway me one way or another.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    That Anonymous Coward (profile), Sep 11th, 2012 @ 5:13pm

    Choices...

    We were stupid and got hacked.
    vs
    We are just inept at running our systems.

    There is no real good way out of this, the upside is they can claim they weren't hacked so they look like the good guys... except they screwed their customers long and hard all on their own then...

    Another reason to not use GoDaddy.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      Chronno S. Trigger (profile), Sep 11th, 2012 @ 5:16pm

      Re: Choices...

      At least they had the balls to admit they screwed up, that's something. They could have said they got hacked, but their security kicked in and shut everything down before data could get stolen. That way they wouldn't have to do any actual investigation and they look like they have advanced security in place.

       

      reply to this | link to this | view in chronology ]

      •  
        icon
        That Anonymous Coward (profile), Sep 11th, 2012 @ 6:20pm

        Re: Re: Choices...

        They still don't have to do any actual investigation.
        We have a lone hacker taking credit vs GoDaddy, I do not like my odds with either side.

        It reminds me of the Sony hacks, where Sony kept claiming they were never breached. They were doing stuff on their own, and only after the evidence reached stupid proportions did they finally admit they were hacked. Then they "found" a single text file with a well known motto to shift the blame to a group that never took credit for the hack. Then it came out how some of the hacks happened was because they skipped over even basic security concepts, and ignored people pointing out glaring flaws in the system.

        Hacks in general happen, we all know this, but it seems to admit to them is worse than being hacked.
        The only secure computer is one not connected to the net and not allowed human interaction.
        If they came out and admitted we got hacked, and we have fixed the problem most people wouldn't think any less of them. It would be a good reminder to stay up to date with your security, and never just assume your secure.
        Instead we get PR speak trying to save the company's reputation with doublespeak and hedging around the issues. Companies want to treat getting hacked as a corporate secret that no one can ever know about, and all that does it make us all less secure.

         

        reply to this | link to this | view in chronology ]

        •  
          identicon
          David, Sep 11th, 2012 @ 8:11pm

          Re: Re: Re: Choices...

          Is there any possibly better way to approach this, though, other than (arguably) admitting a (possible) screw-up and taking flak still?

           

          reply to this | link to this | view in chronology ]

          •  
            icon
            That Anonymous Coward (profile), Sep 12th, 2012 @ 1:11am

            Re: Re: Re: Re: Choices...

            You add more to the mix, like the simple fact they are a massive registrar.

            And then someone over at ars mention in the comments of the story that a DDOS started the problem, then it all went sideways, and they are playing buzzword bingo to make it look like a little booboo from inside rather than the outside.

            I take it with a grain a salt, because on the internet no one knows your a dog... but lots of people are suggesting this doesn't add up as explained unless they were not following even basic rules. Sort of like that cert company who claimed long and hard they weren't hacked, even as people were pulling up bogus certs generated from their systems.

             

            reply to this | link to this | view in chronology ]

  •  
    identicon
    TDR, Sep 11th, 2012 @ 5:18pm

    GoDaddy? How about NoDaddy? One would have to have a double lobotomy to willingly use their service these days.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 11th, 2012 @ 5:45pm

    It was not a hack...because GoDaddy said so. And that's the bottom line!

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 11th, 2012 @ 6:06pm

    Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it's an incredible hard job to isolate the source and to reset each part of the network.

    GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.

    I know you want to kick them because you hate them and all, but come on Mike, can't you be reasonable and accept that shit happens sometimes?

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      That Anonymous Coward (profile), Sep 11th, 2012 @ 6:28pm

      Re:

      Please name the names, times, and dates of the last 5 hosts who went dark because of a cascade router failure.

      Mike did not kick them, Mike pointed out a news story involving *gasp* tech and GoDaddy's statement that says it was not a hack.

      Mike wondered how a mistake like this can happen in a company that is so big, to be kicking them he would have written a piece being much more critical of them and pointing out the massive amount of fail and underhanded tactics they employee. This was merely commentary on a single event and the "response" by GoDaddy.

       

      reply to this | link to this | view in chronology ]

    •  
      identicon
      abc gum, Sep 11th, 2012 @ 6:29pm

      Re:

      "Cascade router failures can happen to almost anyone"

      Plan B? - What's that?


      "Quite simply, it happens"

      Interesting that it happens so little elsewhere, maybe other networks are designed for fault tolerance. Naaa - that can't be it.


      "I know you want to kick them because you hate them and all"

      Awww, isn't that special.

       

      reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, Sep 11th, 2012 @ 7:12pm

      Re:

      Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it's an incredible hard job to isolate the source and to reset each part of the network.

      GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.

      I know you want to kick them because you hate them and all, but come on Mike, can't you be reasonable and accept that shit happens sometimes?


      Please stop disrupting the Techdirt narrative. Any company that supported SOPA is evil, incompetent, should be boycotted and their services suck. Masnick is in a blind rage because after all of his self-congratulations about how he somehow had a stake in defeating SOPA, the end result was a big zero was all of the SOPA players side stepping and using private agreements to accomplish much the same thing.

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        Anonymous Coward, Sep 11th, 2012 @ 7:59pm

        Re: Re:

        You're obsessed with pushing this point on, aren't you?

        Would publicising, legalising, and setting in stone private agreements to leverage further harsher and narrow laws have been an improvement?

         

        reply to this | link to this | view in chronology ]

      •  
        identicon
        abc gum, Sep 12th, 2012 @ 4:40am

        Re: Re:

        Are you as dumb as you sound?

         

        reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, Sep 11th, 2012 @ 8:35pm

      Re:

      I actually agree with this poster. It feels like TechDirt badly wanted to make hay out of this news, and the take of "Which is worse?" feels ham-handed.

      PS, I hate GoDaddy too!

       

      reply to this | link to this | view in chronology ]

    •  
      icon
      Brad C (profile), Sep 11th, 2012 @ 10:53pm

      Re:

      "Cascade router failures can happen to almost anyone."
      "GoDaddy is not immune to the problems of having a sizable network."

      That's the thing, they shouldn't have just one network. When you're making something at this scale one of the design considerations is fault tolerance. The way to achieve that is by designing the system to be separate parts that do not rely on each other. No single change should EVER be able to affect all of their DNS infrastructure all at once like this, it should always require changing it in at least 2 places (for a global DNS infrastructure I'd require at least 6: one per continent, and we'll risk all the customers in Antarctica by lumping them in with someone else)

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        abc gum, Sep 12th, 2012 @ 4:46am

        Re: Re:

        Please stop disrupting the troll narrative. Any post that criticized SOPA supporting corporations is evil, wrong and should be berated. SOPA supporting trolls are in a blind rage because after all of their self-congratulations and gloating about how they will eventually rule the world, the end result was a big zero as all of the SOPA players got bitch slapped and decided to side step the law and use private agreements to accomplish much the same thing.

         

        reply to this | link to this | view in chronology ]

  •  
    icon
    Ninja (profile), Sep 12th, 2012 @ 4:20am

    I know, it's because SOPA didn't pass. Er...

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 12th, 2012 @ 4:32am

    Man I've heard some bull, but this one is really good. Huge operation like GoDaddy and no backup routers. Looks like a lot of people are going to get fired today!

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Sep 12th, 2012 @ 4:37am

    GoDaddy's been hacked for years. There are all kind of back doors that let not reputable people put directories on peoples sites. I know from experience. One day I was editing my site and found a directory with an web page containing an login form for a bank. I didn't put it there and when I told GoDaddy they said it was my fault. I needed to change my password more. Out of there, haven't been hacked since.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    quawonk, Sep 12th, 2012 @ 5:13am

    I believe you Godaddy. Just as much as I believe that you are genuinely against SOPA and not just for PR reasons from the boycott that didn't change shit since apparently people are still using Godaddy.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Gwiz (profile), Sep 12th, 2012 @ 9:21am

    MSM Spin

    I personally really don't care about GoDaddy, but the really intreasting thing in all this I saw was the brief mention on Good Morning America (ABC/Disney).

    Josh Elliot mentioned the story while doing the news yesterday morning. I can't find a link.

    Basically he said that GoDaddy had been taken down reportedly by the internet group Anonymous because they were angry about them supporting laws that could stop them from pirating movies and stuff.

    Woohoo Disney! Way to spin a news report, even when you don't even have ANY of the facts!

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Gwiz (profile), Sep 12th, 2012 @ 9:56am

    Oof. I might have misspoke. I found the video for GMA's yesterday episode and couldn't find the mention. I think I may have mixed up coverage from my local ABC affiliate with the GMA coverage and the local newscaster was the one who stated it like that. Of course the local affiliate doesn't put the full newscasts online, so I won't ever find it now.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, Oct 5th, 2012 @ 12:48pm

    Offical RFO from GoDaddy

    GoDaddy

    Basically, eBGP redistribution into iGP crashed the routers due to a bug in the OS. Due to customer traffic and routing traffic continually overloading the routers, they had to limit the traffic and delay recovery over a period of time causing the extended outage.

    It's a pretty detailed RFO for end-users, and I give them credit for supplying such details.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Josh Gree, Apr 29th, 2013 @ 10:02am

    downtime

    My website and business lives and dies by being available to my customers. DNS Services dnssvc.com offers a product that's so efficient at making sure my website doesn’t go down with a price to match. I will never leave my website unprotected again.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Godaddy Reseller Hosting, Jun 27th, 2013 @ 10:24pm

    Are you as dumb as you sound?

     

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This