Detailed Report Shows How ISPs Are Making 'Business Choice' To Make Your Internet Connection Terrible

from the consumers-lose dept

A couple of years ago, we wrote about an effort by the big broadband players to push the FCC away from using M-Lab to measure basic network diagnostics on the internet. M-Lab is a very interesting project, focused on collecting a huge amount of data about internet performance, and making that data widely available. In the past, for example, we've highlighted an M-Lab project showing which ISPs were throttling BitTorrent.

Now, M-Lab has released a new report, along with all of the data and a very nice tool to analyze it all, called the Internet Observatory, that looks at ISP interconnection and, most importantly, its impact on consumer internet performance. The end result? You guessed it. M-Lab found that the various ISPs have plenty of capacity, but it appears that they make the business decision to let your internet get clogged. No wonder the big telcos wanted the FCC to move away from supporting M-Lab!

The idea that broadband players would leverage interconnection bottlenecks for a business advantage (at the expense of subscriber connections) isn't a new idea. But now there's (lots and lots of) data to confirm that.

The report looks at New York City, where M-Lab was able to collect a ton of data from a variety of different vantage points. What it found was that starting in the spring of 2013, customers on Comcast, Time Warner Cable and Verizon basically choked off content coming from Cogent (one of the major transit providers) every day from the early afternoon until late at night. Those big ISPs would like to try to blame the transit providers for the clogging, but M-Lab's data shows that's not the case. That's because a fourth ISP in the area, Cablevision, showed no similar degradation at all -- showing that Cogent appeared to have plenty of bandwidth available. Rather, as we've suggested in the past, the big broadband players were purposely letting the connection between Cogent (and other transit ISPs) wither on the vine, rather than opening up new ports (a trivially simple process).

Similarly, M-Lab's data shows that the problem wasn't a lack of bandwidth on the part of the ISPs (i.e., no actual technical congestion), because those same ISPs had no problem connecting to a different transit provider, Internap. So the only logical culprit was the interconnection points. There was more than enough bandwidth to go around. There's just the single bottleneck of the interconnection border router (which, again, is trivially simple to get rid of by opening up more ports). Here are the charts showing all of this. First up, you can see the rather notable sluggishness found between Comcast, Time Warner Cable and Verizon on downloading content via Cogent:
And here's what happens from Cablevision users get content via Cogent over that same period. Notice anything different? Oops. Looks like the problem isn't with Cogent's capacity:
And, finally, here's the connection of all four broadband providers to another transit provider, Internap, showing that it's not a problem of capacity with the broadband providers themselves. They all seem fine:
From this, it seems pretty darn clear that the location of the problem is the interconnection with Cogent between Comcast, Time Warner Cable and Verizon.

And the impact of this business decision by those ISPs to let the border router get clogged is massive for end users. Basically, everyone's broadband connections slowed down to a level no sane person would describe as broadband.
While daily median download throughput overall hovered around 4 Mbps, performance degradation was much worse during peak use hours. For much of the time between Spring 2013 and March 2014, download speeds during peak use hours remained well below 4 Mbps. By January 2014, the download throughput rate during peak use hours for Comcast and Verizon traffic over Cogent’s network was less than 0.5 Mbps, the minimum rate necessary for web browsing and email according to the FCC (Figure 4). Note that only between 2:00 AM and 1:00 PM were the three affected Access ISPs (Comcast, Time Warner Cable, and Verizon) able to attain speeds above 4 Mbps across the Transit ISP Cogent. During peak use, only Cablevision achieved sustained rates above 4 Mbps across Cogent, while all Access ISPs achieved sustained rates above 4 Mbps across Internap throughout the day.
As you can see, after about nine months -- right as Netflix gave in and started paying broadband providers to unclog their Cogent connections, things miraculously picked up.

The M-Lab report shows nearly identical situations with Cogent and broadband access providers in other areas like Dallas, LA and Seattle. While the issue with Cogent is the most obvious (perhaps because Netflix is using it and that was the target of the clogging), the report also found those same broadband providers clogging the other major transit provider, Level 3, though not quite as much as with Cogent. As the report notes, there are indications that the degradation of Level 3 was also for business reasons, not any technical reasons. The degradation with Level 3 happened in a nearly simultaneous manner across many different sites "irrespective of their relative baselines." In other words, even when there was plenty of capacity on all sides, it didn't matter. Verizon decided to start clogging Level 3, and boom. Suddenly performance drops. The report found similar "simultaneous" downgrades in download speeds with another large transit provider, XO as well.

The report's conclusions lay things out pretty clearly:
In conducting this research we find a consistent theme across multiple Access ISPs and Transit ISPs: the interconnection relationships between network operators matter. The quality of service that millions of consumers experience on a daily basis for periods of multiple months can be directly tied to these relationships. Further, these relationships are not simply technical. We see the same patterns of degradation manifest in disparate locations across the US. Locations that it would be hard to imagine share any significant infrastructure (Los Angeles and New York City, for example). We thus conclude that the business relationships between impacted Access ISP/Transit ISP pairs is a factor in the repeated patterns of performance degradation observed throughout this research.
Now let's see the various broadband providers and their friends try to spin this as well. And lets see if the FCC finally realizes that interconnection points absolutely are a major issue for an open internet, and that the big broadband players appear to be purposely letting those connections clog for the sake of business negotiations.

Filed Under: broadband, business decision, clogging, interconnection, ports clogged
Companies: cablevision, cogent, comcast, level 3, m-lab, time warner cable, verizon


Reader Comments

The First Word

Subscribe: RSS

View by: Time | Thread


  • icon
    BentFranklin (profile), 29 Oct 2014 @ 10:22am

    I'm writing this from a Verizon connection and I don't have any prob

    reply to this | link to this | view in chronology ]

    • identicon
      jackn, 29 Oct 2014 @ 10:37am

      Re:

      Thanks for the detailed analysis. Based on your comment, I hope that techdirt pulls this article.

      reply to this | link to this | view in chronology ]

    • identicon
      David Linde, 29 Oct 2014 @ 1:48pm

      Re:

      shill (noun) - an accomplice of a hawker, gambler, or swindler who acts as an enthusiastic customer to entice or encourage others.

      reply to this | link to this | view in chronology ]

    • icon
      Sheogorath (profile), 30 Oct 2014 @ 9:46am

      Re:

      From the article: By January 2014, the download throughput rate during peak use hours for Comcast and Verizon traffic over Cogent’s network was less than 0.5 Mbps, the minimum rate necessary for web browsing and email according to the FCC (Figure 4).
      When you comment on this site, that is counted as browsing (according to EE, at least), so of course you wouldn't have any issue with Comcast. Comment again once you've failed in watching Nicki Minaj's latest video on YouTube.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 29 Oct 2014 @ 10:40am


    Now let's see the various broadband providers and their friends try to spin this as well. And lets see if the FCC finally realizes that interconnection points absolutely are a major issue for an open internet, and that the big broadband players appear to be purposely letting those connections clog for the sake of business negotiations.


    The weak and corrupt FCC won't do anything about it, and broadband providers are never going to willingly open up on their own. At this point we really only have one, last hope: Google Fiber.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 29 Oct 2014 @ 12:06pm

      Re:

      Obi Wan Kenobi

      reply to this | link to this | view in chronology ]

    • identicon
      alternatives(), 29 Oct 2014 @ 12:45pm

      Re:

      one, last hope: Google Fiber.

      Meet the new boss, same as the old boss.

      Why is the "hope" a Corporation? If land is going to be grabbed for right of ways, why not a co-op?

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 29 Oct 2014 @ 1:45pm

        Re: Re:

        Using an ideal scenario to argue against the lesser of several evils that is proportionately much better than those other evils is a good way to never improve the system. Let's get Google Fiber and then talk co-op. Some municipalities will never be able to overcome their local corruption and bureaucratic obstacles, so Google Fiber or a competitor who ups their game due to Google Fiber's availability may be the only way forward for some communities.

        reply to this | link to this | view in chronology ]

      • icon
        PaulT (profile), 30 Oct 2014 @ 2:09am

        Re: Re:

        "Why is the "hope" a Corporation?"

        Why not both? Realistically, Google seem to be making more headway with local governments than smaller municipal schemes, and they're the ones capable of installing new infrastructure. Therefore, they're best placed to break existing monopolies. Once the monopolies start to crumble, then there's room for others to move in, but you may need the elephant to get the ball rolling.

        reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 29 Oct 2014 @ 10:47am

    Well, yes, it is a business decision. The thing is, you move your peering relationships among different transit providers to obtain the best price. Tier 1 ISP A may charge X per gigabit at the 95th percentile while T1 ISP B charges Y. And they change all the time.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 29 Oct 2014 @ 10:57am

      Re:

      The clogging wouldn't have anything to do with Netflix killing the other half of the ISPs business, their cable offering? The ISPs wouldn't be trying to get Netflix to pay them to make up that lost income? Doing so would be abuse of an effective monopoly position, as most customers only have the choice of one broadband provider.
      The ISPs wouldn't abuse their monopoly position like this would they?

      reply to this | link to this | view in chronology ]

      • identicon
        any moose cow word, 29 Oct 2014 @ 11:08am

        Re: Re:

        It's not just one of the biggest ISPs but three playing this game. If you do have a choice between providers, you're still likely to be screwed.

        reply to this | link to this | view in chronology ]

      • identicon
        PRMan, 29 Oct 2014 @ 3:46pm

        Re: Re:

        Netflix offered to put hardware containing their most popular streams AT the ISP in each of these cases for free. And they were refused.

        reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 29 Oct 2014 @ 11:25am

      Re:

      When you are talking about Comcast, Verizon, and such most likely these are non-paid peering links based on some unknown agreement... That's typically the definition of a Tier 1 ISP in all reality. Sure they pay for cross connects at times in a place like Equinox, TelX, et al and the physical hardware, but generally little money ever changes hands.

      The business decision is usually in the form of ISP A saying that Cogent is breaking the initial agreement and wants some sort of kick back, while Cogent is saying that Comcast is the one breaking the agreement and needs more paid links. (usually an ISP will have both even to the same provider)

      At the end of the day however it's still the ISP's job to deliver the services that their customers are paying for, so they have to either pony up or route around it.

      reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 31 Oct 2014 @ 2:02am

      Re: diatribe

      Nice diatribe. The question is where to balance the money earned. Does it go to investors, wire construction companies,peering or bonuses to the executives. PG&E in Northern California found it profitable to send the money to bonuses until a gas pipe blew up and several forest fires were attributed to lack of tree trimming maintaining.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 29 Oct 2014 @ 11:10am

    That first linked chart makes it appear like collusion is in play as well...

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 29 Oct 2014 @ 12:17pm

    The ONLY thing that would work

    is that someone forcefully investigated these companies from top to bottom, and if (yeah I know... there really doesn't seem to be any doubt left) it is proven to be willful degradation of customer's broadband, they should pay every cent of that lost internet connection back to the customer.
    But the speed during peak hours is worth much more than speed at night, so there would have to be some calculation done as to worth of connection compared to time of day. With these numbers I would think some place between 60-70% refund.
    It would be billions and that is without the MAJOR fine on top. They would listen to that.
    But alas, this is but an impossible dream that can never come true.

    reply to this | link to this | view in chronology ]

  • icon
    The Wanderer (profile), 29 Oct 2014 @ 12:23pm

    Anyone else find it amusing that 0.5 Mbps (512 Kbps, or 64 KB/s) is considered the minimum speed necessary for Web browsing and E-mail, when for so many years the top speed available (without paying ridiculous premiums) was ~53KB/s?

    Not that it isn't necessarily a reasonable minimum nowadays - but if it is, that says something unfortunate about the footprint of modern Websites... to say nothing of what people think of as reasonable in E-mail nowadays.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 29 Oct 2014 @ 2:02pm

      Re:

      WWIV rocked even on 28.8...
      Damn whippersnappers just don't know what good is.

      Seriously though WordPress, Drupal, and other content management systems usually add so much fluff to the webpages and added calls to other sites like JSON usually without compression that things really have gotten out of hand. Check out this site, http://www.webpagetest.org, for some good optimization tips.

      reply to this | link to this | view in chronology ]

    • icon
      PaulT (profile), 30 Oct 2014 @ 2:17am

      Re:

      Whether you're talking about disk space, RAM or bandwidth, nobody cares about optimising for the lower end since those things are so abundant and cheap. The days of squeezing every last byte of efficiency out of your code are long gone.

      reply to this | link to this | view in chronology ]

      • icon
        Ninja (profile), 30 Oct 2014 @ 4:36am

        Re: Re:

        I do agree but I also think we should optimize while there is no scarcity to avoid scarcity. See mobile connections for instance. I have 75mb per week to use in one of my phones and 200 in the other one (I won't get more because mobile connections are so crappy that they barely serve any purpose here) so in my case any optimization is welcome. Sure it's an artificial limit but considering the crapiness overall it would still help a lot even without the caps.

        reply to this | link to this | view in chronology ]

  • identicon
    alternatives(), 29 Oct 2014 @ 3:38pm

    Anyone else find it amusing that 0.5 Mbps (512 Kbps, or 64 KB/s) is considered the minimum speed necessary for Web browsing and E-mail, when for so many years the top speed available (without paying ridiculous premiums) was ~53KB/s?

    Amusing as in ha-ha, interesting, or some other form of amusement?

    The 53Kb/s sites lacked javascript, heavy graphics, or sound/motion pictures.

    Today's web sites allow the "marketers" who want movement and noise to have that on a web site.

    reply to this | link to this | view in chronology ]

  • icon
    Manabi (profile), 29 Oct 2014 @ 4:49pm

    It affected users outside of those areas too

    I ran into this myself last year. I have a server that my route to it from Comcast went through Cogent. And during peak hours it was dial-up speeds. In the graph below I was doing backups continuously for nearly 24 hours straight (there's a gap in there where one finished and I hadn't started the new one yet). The graph matches the stuff MLab found:

    Graph

    As you can see, from about 6pm - 2am my connection to my server was basically unusable. It was so bad it would affect ssh! I would get very noticeable typing lag in my ssh sessions.

    Once Netflix caved and paid Comcast, it got better immediately and is now back to normal.

    reply to this | link to this | view in chronology ]

  • icon
    Ninja (profile), 30 Oct 2014 @ 3:15am

    More damning evidence that the ISPs can't be trusted and the FCC has to step in. Title II seems more and more reasonable each passing day.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 30 Oct 2014 @ 3:42am

    If not even a slap to the hand, and an immediate stop to the practice......then nothing has changed, trust in gov will still be zero, as their priorities will be inline with corporations legally having monopolies, less choice to consumers, better control over mediums(company:customer ratio)

    reply to this | link to this | view in chronology ]

  • identicon
    anonymous, 31 Oct 2014 @ 11:03am

    No support for allegations Netflix was deliberately throttled

    This article should have been titled "Detailed Report Provides No Support For Allegations Netflix Was Deliberately Throttled. Interconnect Congestion is Commonplace."

    reply to this | link to this | view in chronology ]

    • icon
      That One Guy (profile), 1 Nov 2014 @ 7:35pm

      Re: No support for allegations Netflix was deliberately throttled

      I'm guessing you didn't bother to look at the graphs shown, or read the article. If you had, you might have noticed this little tid-bit:

      As you can see, after about nine months -- right as Netflix gave in and started paying broadband providers to unclog their Cogent connections, things miraculously picked up.

      Nine months of 'network congestion', and then, as though through a miracle, just as Netflix starts paying them more, suddenly the network cleared right up. You'd have to be blind, either accidentally or intentionally, not to see the link there.

      reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown for basic formatting. (HTML is not supported.)
  • Remember name/email/url (set a cookie)

Close

Add A Reply

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown for basic formatting. (HTML is not supported.)
  • Remember name/email/url (set a cookie)

Follow Techdirt
Techdirt Gear
Shop Now: Copying Is Not Theft
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.