by Karl Bode

Filed Under:
broadband, david raphael, fcc, net neutrality

at&t, comcast, netflix, verizon, youtube

Netflix, YouTube Streaming Gets Worse on Major ISPs, Who In Turn Blame the Deep, Dark Mysteries of the Internet

from the what-the-hell-is-going-on dept

Last week this blog entry by iScan developer David Raphael saw a lot of attention as proof positive that Verizon was violating network neutrality. Specifically, Raphael observed that services using Amazon Web Services accessed from his home FiOS connection (including Netflix) delivered bits more slowly than services accessed on his business FiOS connection. As his supposed nail in the neutrality-violation coffin, Raphael provided a chat transcript from a Verizon rep supposedly "confirming" that the company was throttling select services. This "proof" then bumbled and stumbled its way around the Internet as evidence neutrality had been trampled.

Except as anybody who has spent any time on the phone with a support rep knows, what comes out of their mouths may or may not have any direct tie to this particular plane of reality. It's also worth noting that the kind of Netflix stuttering Raphael saw has been going on for much of the last year across numerous ISPs, long before Verizon's recent court victory over the FCC's neutrality rules. Verizon also denied they were manipulating traffic in a statement sent to all media outlets:
"We treat all traffic equally, and that has not changed. Many factors can affect the speed a customer's experiences for a specific site, including, that site’s servers, the way the traffic is routed over the Internet, and other considerations. We are looking into this specific matter, but the company representative was mistaken. We are going to redouble our representative education efforts on this topic."
While Raphael may have stumbled into Verizon shenanigans, what he saw could also be explained by normal peak network or peering congestion, and we've long discussed the problems with crying net neutrality wolf. Peering relationships are indeed complicated, with ISPs, content companies, and Internet backbone operators all engaged in an endless tug of war over video profits and peering imbalances (I recommend this piece last summer from Jon Brodkin at Ars Technica or this piece from last week by GigaOM's Stacey Higginbotham).

With that as background, fast forward to this week when Netflix released their latest ISP streaming rankings, which show that most of the nation's biggest ISPs have been dropping in regards to their Netflix streaming performance. ISPs that partner with Netflix's Open Connect CDN do better in the rankings, and major ISPs have tried somewhat feebly to claim Netflix's attempt to name and shame them into using their CDN is a sort of reverse network neutrality.

Still, Verizon FiOS (and DSL), Comcast, and AT&T's U-Verse services have seen major drops in the rankings not entirely explained by their refusal to use Netflix's CDN. Being large network operators, you would imagine that they'd be able to explain what exactly is happening to stink up millions of broadband users' Netflix experiences, though Comcast, AT&T and Time Warner Cable refuse to comment to the media on their collective performance nose dive. Netflix also refuses to comment. Verizon, with all of their network expertise, continues to blame the problems on either unknown Internet bogeymen or Netflix:
"We state unequivocally that Verizon’s broadband Internet access services deliver a pristine user experience to our customers at any time of day on every day of the week. This has been repeatedly proven through independent testing by the Federal Communications Commission (FCC), which has conclusively demonstrated that FiOS Internet consistently delivers both download and upload speeds in excess of what we advertise. In short, our Internet customers often get more than they pay for. How the Internet works can be complicated, and consumers should be aware of the fact that the integrity of their home Internet connection is only a portion of the streaming video quality equation. If their broadband connection is functioning correctly, the source of their frustration and the content they wish to see may be one in the same."
In summation, the only major ISP wiling to comment on why millions of people are having YouTube and Netflix issues on fast connections insists "it's complicated." So what's really happening? One theory du jour is that companies like Verizon are abusing their gatekeeper position and letting their peering points to companies like Netflix saturate in order to gain leverage in negotiations and rhetoric, while giving a leg up to their own CDN or services (like Red Box Instant by Verizon). Verizon's certainly no stranger to these kinds of tactics, but again, there's not enough data to prove that hypothesis, and while neutrality rules are dead, big ISPs aren't exactly eager to do anything to invite new neutrality regulations.

The reality is that despite the best guesses of some very smart people, the jury is out because we lack enough data to make full conclusions just yet. The good news is we should see some more robust data in the coming months as data from collection operations like MLAB comes to light in looming FCC ISP performance reports. In addition, YouTube has started ranking ISPs based on YouTube streaming performance, which will also hopefully shed a little more light on precisely which Internet bogeymen are to blame for your stuttering video feed.

Reader Comments (rss)

(Flattened / Threaded)

  1. identicon
    Anonymous Coward, Feb 11th, 2014 @ 2:30pm

    did you expect them to own up and openly say that they are not throttling, they have just reduced the available bandwidth allotted to customers using Netflix!

    reply to this | link to this | view in thread ]

  2. identicon
    Anonymous Coward, Feb 11th, 2014 @ 2:34pm

    I'd be willing to bet its some kind of routing issue.

    I use Verizon and have seen absolutely ludicrous IP routing that often leads to drastically reduced speeds.

    Once while transferring a file between my house and a friend about 30 miles across town, I noticed extremely low speeds (we both have 50+ Mbps normally). I tried a trace-route and saw packets being routed through Atlanta, Washington D.C. and Chicago... and we are both in Florida... data was being sent over thousands of miles to travel between two computers in the same city.

    reply to this | link to this | view in thread ]

  3. icon
    TKnarr (profile), Feb 11th, 2014 @ 2:39pm

    This may backfire on the ISPs

    Users don't buy Internet service for the ISP. They buy it for everything else out there. The ISP's service is just the pipe as far as most people are concerned. If the ISPs degrade service too far, people will start looking for another ISP to get their pipe from.

    The major choices for broadband here in San Diego are Cox, Time Warner and AT&T. TW and AT&T ought to worry that, when looking at houses I could possibly buy, one of my criteria is "located in an area serviced by Cox" because I just don't want to deal with the ongoing headaches I'm sure to have with the other two.

    reply to this | link to this | view in thread ]

  4. icon
    Karl Bode (profile), Feb 11th, 2014 @ 2:40pm

    Netflix has commented

    Worth adding this investor analyst note that claims Netflix doesn't believe Verizon is intentionally hampering their streams:

    so far, that’s not happening, Netflix says.

    That update comes to us via a note from J.P. Morgan analyst Doug Anmuth, who says he has been talking to Netflix CEO Reed Hastings and CFO David Wells, and they told him they don’t think cable and telco companies are hampering the company’s video streams.

    Anmuth doesn’t have much to report on the topic, so here are his comments in their entirety: “Netflix does not seem overly concerned regarding Net Neutrality, and continues to believe that violations would be escalated quickly. Netflix also indicated that it has no evidence or belief that its service is being throttled.”

    reply to this | link to this | view in thread ]

  5. icon
    DannyB (profile), Feb 11th, 2014 @ 2:44pm


    It may be a routing issue.

    But that doesn't mean it isn't deliberate.

    reply to this | link to this | view in thread ]

  6. identicon
    Baron von Robber, Feb 11th, 2014 @ 2:44pm

    There is a way to tell...

    Get a VPN service.
    Try it with and without.
    If it's bout the same, they are not degrading Netflix/Youtube.

    If there is a difference, they are.

    reply to this | link to this | view in thread ]

  7. identicon
    Anonymous Coward, Feb 11th, 2014 @ 2:56pm

    Re: Re:

    Makes sense...

    Verizon's routers likely automatically send "Low priority" traffic on the least efficient route.

    Their packet inspector probably mistook my ssh session for bittorrent or something.

    reply to this | link to this | view in thread ]

  8. identicon
    Anonymous Coward, Feb 11th, 2014 @ 2:57pm

    Re: This may backfire on the ISPs

    You have one maybe two ISPs in most areas and with places like Kentucky pushing legislation to make city/county based ISPs illegal, or extremely difficult to establish or continue on, there really isn't anywhere else to go. Its like politics, you got a red pile of crap and a blue pile of crap, and no matter the color you have a pile of crap.

    reply to this | link to this | view in thread ]

  9. identicon
    Anonymous Coward, Feb 11th, 2014 @ 2:58pm

    Re: There is a way to tell...

    What if they also degrade VPN connections?

    reply to this | link to this | view in thread ]

  10. identicon
    Baron von Robber, Feb 11th, 2014 @ 3:02pm

    Re: Re: There is a way to tell...

    That would be messed up. VPN does have overhead so all things being equal, there would be a little slowdown.
    But to throttle that as well?

    Time to go back to dialup =b

    reply to this | link to this | view in thread ]

  11. identicon
    Baron von Robber, Feb 11th, 2014 @ 3:04pm

    Re: Re: Re: There is a way to tell...

    Forgot to mention, over on Ars in the comment section on the same topic, some have tried this and miracously, with VPN, their Netflix/Youtube got faster. :/

    reply to this | link to this | view in thread ]

  12. identicon
    Anonymous Coward, Feb 11th, 2014 @ 3:10pm

    Re: Re: Re: Re: There is a way to tell...

    If VPN traffic goes up they will start forwarding it through the NSA super computer to decrypt the packets and determine if you're using in to get Netflix and then throttle all of your connections to 128kbps

    reply to this | link to this | view in thread ]

  13. identicon
    Anonymous Coward, Feb 11th, 2014 @ 3:13pm

    Re: Re: Re: Re: There is a way to tell...

    Honestly, I would not be surprised if that is due to routing.

    As per my previous post above, bad routing can introduce serious connection slowdown. By going through a VPN, assuming you have a decent connection to it, you can bypass misconfigured or malicious routers ISP side.

    Its not entirely unreasonable for an ISP to use bad routing as a way of throttling. That way they can claim to not be "throttling" a connection, and they are technically not... they just route your packets to mars...

    reply to this | link to this | view in thread ]

  14. identicon
    Anonymous Coward, Feb 11th, 2014 @ 3:13pm

    Re: Re: Re: Re: Re: There is a way to tell...

    To fight terrorisms and such

    reply to this | link to this | view in thread ]

  15. identicon
    Anonymous Coward, Feb 11th, 2014 @ 3:14pm

    bullshit. the ISPs are deliberately throttling youtube. here in germany it is known fact. Telekom, the biggest provider does deliberately reduce the bandwith because some "issues" with peering agreements, meaning telekom wants to be paid extra for that stuff on top of the regular agreements. Of course that doesn't fly and the customers have to suffer for it.

    so yeah, no mystery here, just plain greed.

    reply to this | link to this | view in thread ]

  16. icon
    Mark Gisleson (profile), Feb 11th, 2014 @ 3:19pm

    More anecdotal evidence

    I moved to a remote rural location with no cable or dish options available. I had to set up a Verizon Home Fusion device. It is amazing. It can download the Library of Congress in about half an hour. OK, 30mbps isn't that fast, but at $5 a GB I had to dump my Netflix account because I couldn't afford the bandwidth it used.

    That and even at 30 mbps, Netflix was choppy and unwatchable. All other streaming video was better, including illegal NBA feeds.

    State of the art wireless technology, but it couldn't stream Netflix worth a crap.

    There's something else going on here.

    reply to this | link to this | view in thread ]

  17. icon
    RonKaminsky (profile), Feb 11th, 2014 @ 3:20pm

    Re: Re: Re:

    Hmm, "ssh"... "Washington D.C."...

    Brings to mind: NSA. You probably share an ISP and similar location to someone who's connected to someone who's suspicious, eh?

    reply to this | link to this | view in thread ]

  18. icon
    Karl Bode (profile), Feb 11th, 2014 @ 3:24pm

    Re: More anecdotal evidence

    I've been on FiOS for five years, and about 50% of the Time YouTube streams are entirely unwatchable.

    reply to this | link to this | view in thread ]

  19. icon
    Gabriel J. Michael (profile), Feb 11th, 2014 @ 3:24pm


    I also love the bit in Verizon's response about how since speed tests show people getting higher than advertised speeds, their customers are getting "more than they paid for" and presumably should STFU.

    It's like saying that since you can go 80 mph on the Beltway at 3 AM, you shouldn't complain about traffic.

    reply to this | link to this | view in thread ]

  20. identicon
    Anonymous Coward, Feb 11th, 2014 @ 3:26pm

    Re: Re: Re: Re:

    Oh noes!

    reply to this | link to this | view in thread ]

  21. identicon
    Anonymous Coward, Feb 11th, 2014 @ 3:28pm

    It would seem that much of Netflix's success was a result of the actions of ISPs' -- by throttling Bittorrent through both active packet-shaping and by extremely restricting the upstream bandwidth.

    If restricting Bittorrent traffic was justified by the ISPs for business reasons, why should those same reasons not apply to Netflix, which is a much bigger bandwidth hog?

    reply to this | link to this | view in thread ]

  22. icon
    John Fenderson (profile), Feb 11th, 2014 @ 3:36pm

    You have choices?

    At my place, my choices for broadband ISPs consist of Comcast, Comcast, and Comcast.

    Consider yourself very lucky indeed!

    reply to this | link to this | view in thread ]

  23. identicon
    Mark Milliman, Feb 11th, 2014 @ 3:38pm

    Traffic Equality

    The article states the problem very clearly. The problem is that all traffic is treated equal. For better or for worse, all traffic IS NOT EQUAL. Voice is more sensitive than video that is more sensitive than all else. Latency and jitter is increased in port queues that will impact the end performance of a service. By treating all traffic the same your voice and video is bound to be impaired no matter how much bandwidth is thrown at it. As long as you people insist that Net Neutrality means treating all traffic the same, you will be eternally searching for the boogie man.

    reply to this | link to this | view in thread ]

  24. identicon
    Beech, Feb 11th, 2014 @ 4:46pm


    If Verizon didn't want people thinking they were violating net neutrality, they shouldn't have fought a lawsuit over their rights to violate net neutrality. They opened Pandora's Box, and now every time one of their customers has a hiccup in internet speed they're going to blame it on throttling. If Verizon really cared so much about "treating all traffic equally" they should have just settled the lawsuit by saying "okay". Problem solved.

    reply to this | link to this | view in thread ]

  25. identicon
    AnonCow, Feb 11th, 2014 @ 5:13pm

    I trust Verizon and Comcast with packets like I trust Jamie Dimon with my grandmother's life savings.

    reply to this | link to this | view in thread ]

  26. identicon
    Anonymous Coward, Feb 11th, 2014 @ 5:22pm

    Re: Re: Re: There is a way to tell...

    Take a close look at the terms of service your ISP gives you. I don't think it's been enforced all that often, but some of them don't allow VPN connections on residential accounts.

    So if that becomes a common solution to video streaming, degrading them might not be necessary.

    reply to this | link to this | view in thread ]

  27. identicon
    Anon77, Feb 11th, 2014 @ 5:24pm

    Re: Re: Re: Re: Re: There is a way to tell...

    Route all traffic over a peering point that cant handle that much traffic and you have effectively throttled the traffic. Yet you can say it's not throttled as you haven't used your throttling specific hardware.

    They are being crafty.

    virgin media in the UK had terrible youtube streaming for a long time and blamed 3rd party suppliers. These were in fact a local youtube CDN attached to their own network yet they still didn't give it enough bandwidth to run correctly.

    bottom line is never trust an isp, they will screw customers over any way they can

    reply to this | link to this | view in thread ]

  28. identicon
    rapnel, Feb 11th, 2014 @ 5:44pm

    Re: Re: Re: Re: There is a way to tell...

    Not allowing VPN connections on residential accounts would be preposterous. That's a direct attack on your ability to communicate privately and in private. Unacceptable and not only with regards to any supposed terms of service.

    reply to this | link to this | view in thread ]

  29. identicon
    Anonymous Coward, Feb 11th, 2014 @ 6:11pm

    Re: There is a way to tell...

    That's incorrect. Let's say Verizon uses Cogent to get to AWS, and your VPN uses Level3 to get to the VPN point A and it's bandwidth to get to AWS. If Verizon is jamming the link to Cogent, your VPN could have less latency to AWS and better performance.
    The sad fact is, most CDNs like Netflix would gladly peer with Verizon for Free, but it's in the dual providers best interest not to, ie Verizon and RedBox.
    Peering agreements are a lag on the internet, and sadly most network engineers understand this, that is why it's important to peer with as many providers as possible and keep those links from being saturated.

    If you really want to test this out, I would suggest first using mtr with both the vpn and without and seeing where the loss of packets is occuring and high latency points. It's not full proof but with enough data points and also using looking glass sites, you could narrow down the link with the congestion. Given if you are a residential user not announcing BGP with multiple providers, good luck finding someone to listen.

    reply to this | link to this | view in thread ]

  30. icon
    Chronno S. Trigger (profile), Feb 11th, 2014 @ 6:11pm

    The problem with residential connections

    This guy states a few facts that suggest a vary possible reason for the discrepancy.

    Business connections are dedicated connections, you don't share the bandwidth. With a residential connection, you do.

    Then we can probably safely assume that his business testing was done during business hours and home testing was done after. I have found that during the day my connection to Youtube is amazing, 1080p video, but right around 3:30pm every single day I can't watch the 144p videos. It picks back up again after 11:30pm. Took me a little while to figure it out, but it turns out that the connection would start sucking right as the kids started getting home from school. Weekends, summer, and the random snow days didn't have this problem. The kids would come home, play games, watch videos, and kill the internet connection for the entire neighborhood.

    The "Deep, dark mysteries of the Internet" are a damn good excuse because they're true. We're going to need something much more then random ping tests to prove Verizon is throttling.

    reply to this | link to this | view in thread ]

  31. identicon
    Anonymous Coward, Feb 11th, 2014 @ 6:42pm


    A few people have touched on this already, latency is the real killer here not your Mbps.

    Q. Whats faster?
    A) 100Mbps connection with 1.0 Second of latency
    B) 50Mbps connection with 0.5 seconds of latency?

    A. Both are the same.

    How the data is routed will have a huge impact on latency.
    From the data center at work my latency is minimal to nearly everywhere, this is because we are only one hop away from several major providers. (Level3, TW Telecom, XO, Time Warner Cable, AT&T, Hurricane and many more.)

    From home on my cable modem the latency on just my ISP's routers is terrible but oddly the latency to their speed test server is minimal so there is "nothing wrong"

    reply to this | link to this | view in thread ]

  32. icon
    MikeC (profile), Feb 11th, 2014 @ 8:30pm

    Ping of NetFlix.Com -- via VPN/Corporate Network vs Via Comcast Direct within 2 minutes of each other


    7 59 ms 60 ms 60 ms []
    8 135 ms 125 ms 132 ms []
    9 135 ms 141 ms 140 ms []
    10 149 ms 136 ms 131 ms []
    11 126 ms 132 ms 129 ms []
    12 142 ms 141 ms 138 ms []
    13 140 ms 136 ms 133 ms []
    14 136 ms 138 ms 141 ms []


    7 28 ms 30 ms 30 ms []
    8 27 ms 28 ms 26 ms []
    9 84 ms 32 ms 34 ms []
    10 * * * Request timed out.
    11 104 ms 102 ms 102 ms
    12 102 ms 103 ms 101 ms []
    13 102 ms 102 ms 102 ms []

    First 6 entries almost identical just local hops, but when it get's handed off to the backbone it gets interesting. Note the different destinations depending on the backbone when it get to the same IP address but different domains resolved based on the backbone it rode? Not all that surprising but also when I route local via comcast - look at hop 10 when it's handed off to Qwest (not surprising in my experience with Qwest networks).

    The VPN definitely increases the latency across the board as expected. But the fact that netflix is resolving different FQDN's to the same IP from different aggregate ports in their network, a touch weird but then I don't work much with networks with that kind of capacity.

    Ever wonder why they call it a CLOUD?? Fully opaque, easy to get lost, only god/buddha/allah know how it works and they aren't really sure. So as much as I hate big ISP's I never ascribe malice to what most likely is based in incompetence.

    reply to this | link to this | view in thread ]

  33. identicon
    fred, Feb 11th, 2014 @ 9:05pm


    Has anyone considered that perhaps the demand for those services has grown too quickly? Peering is not unlimited, and the amount of total bandwidth cannot increase endlessly, at least not without adding more connections, which costs.

    Also, it depends on every step from provider to destination, and the increases in demand now are likely very much like the early days or torrents. Isps cannot add connectivity that quickly to meet a sudden spike in user demand.

    reply to this | link to this | view in thread ]

  34. icon
    techflaws (profile), Feb 11th, 2014 @ 10:39pm


    Was it justfied? What do consumers like to pay for? Restricted Internet or free?

    reply to this | link to this | view in thread ]

  35. icon
    DannyB (profile), Feb 12th, 2014 @ 6:36am

    Re: Re: Re:

    Maybe it's not a least efficient route. Maybe it's a route that passes through an NSA inspection checkpoint.

    Hey that Netflix traffic is encrypted! Could be terrorists.

    reply to this | link to this | view in thread ]

  36. icon
    DannyB (profile), Feb 12th, 2014 @ 6:40am

    Re: Traffic Equality

    Yes, latency matters. For interactive protocols.

    But good bandwidth is sufficient for video streaming applications like Netflix, YouTube, etc. It doesn't matter if your video stream is seconds behind the packets leaving the server, as long as there is bandwidth for a continuous flow of packets to your screen and speakers.

    reply to this | link to this | view in thread ]

  37. icon
    John Fenderson (profile), Feb 12th, 2014 @ 9:12am

    Re: The problem with residential connections

    Well said.

    There's a million "gotchas" like this. It's premature to accuse anybody of being a bad actor without more comprehensive analysis and testing.

    reply to this | link to this | view in thread ]

  38. identicon
    zip, Feb 12th, 2014 @ 10:34am

    Re: Latency

    Latency means little to downloading/uploading applications that use multiple connections/streams, because when one connection is idly waiting for a packet acknowledgment, the other connections are busy sucking down maximum bandwidth.

    reply to this | link to this | view in thread ]

  39. identicon
    Anonymous Coward, Feb 12th, 2014 @ 1:48pm

    its not about a "major" isp's, .....what do "major" anythings usually have?

    Alot of customers, .....thats why they are most likely the first to do something not in favour of the people, why would the "pressures that be" pressure a ISP with 500 users, when they want to "affect", the 2 million market, the bigger a WELL INTENTIONED company gets, the bigger the pressures they recieve, the harder they have to keep their original course.........and this is no excuse for them to FAIL this, because they have 2million damn customers, that alone means any bad decisions is MAGNIFIED.......i dont just want to support a top quality service, i want to support a MORAL one, can go on and assume that is my damn fucking line in the morals, then piss off with whatever it is your shoveling

    Not had me coffee

    reply to this | link to this | view in thread ]

  40. icon
    John85851 (profile), Feb 12th, 2014 @ 2:50pm

    But they don't say they're not doing it

    Some commenters at Ars make a good point: the statement from Verizon basically says their network is operating as expected, but they don't specifically say they aren't throttling Netflix. Why don't they "state unequivocally" that they're not doing it?

    reply to this | link to this | view in thread ]

  41. identicon
    Anonymous Coward, Feb 12th, 2014 @ 10:53pm

    Re: Re: Re: Re:

    Wow, a lot of people share my IP, because of the way my shared vps provider works. I can confirm we're all evil because of that.

    reply to this | link to this | view in thread ]

  42. identicon
    The Secret Tor Files, Dec 11th, 2014 @ 8:00am

    The Truth

    The sad truth is if your running Netflix or YouTube, you are getting throttled, which you are you can barely get 320p video quality. Yet somehow if you suddenly set up a vpn and run YouTube you can get 1080 without a problem, its now a sad reality that these companies disregard net neutrality for profit.

    reply to this | link to this | view in thread ]

  43. identicon
    Annie lobo, Nov 28th, 2016 @ 1:53am


    I use to stream movies via NetFlix, the video quality is really good but it's pretty expensive for me. Thus, I just went through your write up, it's really useful. I have tried few websites given, the stock of content is amazing to watch on them. Thank you for the help!

    reply to this | link to this | view in thread ]

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Insider Shop - Show Your Support!

Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Report this ad  |  Hide Techdirt ads
Recent Stories
Report this ad  |  Hide Techdirt ads


Email This

This feature is only available to registered users. Register or sign in to use it.