Netflix, YouTube Streaming Gets Worse on Major ISPs, Who In Turn Blame the Deep, Dark Mysteries of the Internet

from the what-the-hell-is-going-on dept

Last week this blog entry by iScan developer David Raphael saw a lot of attention as proof positive that Verizon was violating network neutrality. Specifically, Raphael observed that services using Amazon Web Services accessed from his home FiOS connection (including Netflix) delivered bits more slowly than services accessed on his business FiOS connection. As his supposed nail in the neutrality-violation coffin, Raphael provided a chat transcript from a Verizon rep supposedly “confirming” that the company was throttling select services. This “proof” then bumbled and stumbled its way around the Internet as evidence neutrality had been trampled.

Except as anybody who has spent any time on the phone with a support rep knows, what comes out of their mouths may or may not have any direct tie to this particular plane of reality. It’s also worth noting that the kind of Netflix stuttering Raphael saw has been going on for much of the last year across numerous ISPs, long before Verizon’s recent court victory over the FCC’s neutrality rules. Verizon also denied they were manipulating traffic in a statement sent to all media outlets:

“We treat all traffic equally, and that has not changed. Many factors can affect the speed a customer’s experiences for a specific site, including, that site’s servers, the way the traffic is routed over the Internet, and other considerations. We are looking into this specific matter, but the company representative was mistaken. We are going to redouble our representative education efforts on this topic.”

While Raphael may have stumbled into Verizon shenanigans, what he saw could also be explained by normal peak network or peering congestion, and we’ve long discussed the problems with crying net neutrality wolf. Peering relationships are indeed complicated, with ISPs, content companies, and Internet backbone operators all engaged in an endless tug of war over video profits and peering imbalances (I recommend this piece last summer from Jon Brodkin at Ars Technica or this piece from last week by GigaOM’s Stacey Higginbotham).

With that as background, fast forward to this week when Netflix released their latest ISP streaming rankings, which show that most of the nation’s biggest ISPs have been dropping in regards to their Netflix streaming performance. ISPs that partner with Netflix’s Open Connect CDN do better in the rankings, and major ISPs have tried somewhat feebly to claim Netflix’s attempt to name and shame them into using their CDN is a sort of reverse network neutrality.

Still, Verizon FiOS (and DSL), Comcast, and AT&T’s U-Verse services have seen major drops in the rankings not entirely explained by their refusal to use Netflix’s CDN. Being large network operators, you would imagine that they’d be able to explain what exactly is happening to stink up millions of broadband users’ Netflix experiences, though Comcast, AT&T and Time Warner Cable refuse to comment to the media on their collective performance nose dive. Netflix also refuses to comment. Verizon, with all of their network expertise, continues to blame the problems on either unknown Internet bogeymen or Netflix:

“We state unequivocally that Verizon’s broadband Internet access services deliver a pristine user experience to our customers at any time of day on every day of the week. This has been repeatedly proven through independent testing by the Federal Communications Commission (FCC), which has conclusively demonstrated that FiOS Internet consistently delivers both download and upload speeds in excess of what we advertise. In short, our Internet customers often get more than they pay for.

How the Internet works can be complicated, and consumers should be aware of the fact that the integrity of their home Internet connection is only a portion of the streaming video quality equation. If their broadband connection is functioning correctly, the source of their frustration and the content they wish to see may be one in the same.”

In summation, the only major ISP wiling to comment on why millions of people are having YouTube and Netflix issues on fast connections insists “it’s complicated.” So what’s really happening? One theory du jour is that companies like Verizon are abusing their gatekeeper position and letting their peering points to companies like Netflix saturate in order to gain leverage in negotiations and rhetoric, while giving a leg up to their own CDN or services (like Red Box Instant by Verizon). Verizon’s certainly no stranger to these kinds of tactics, but again, there’s not enough data to prove that hypothesis, and while neutrality rules are dead, big ISPs aren’t exactly eager to do anything to invite new neutrality regulations.

The reality is that despite the best guesses of some very smart people, the jury is out because we lack enough data to make full conclusions just yet. The good news is we should see some more robust data in the coming months as data from collection operations like MLAB comes to light in looming FCC ISP performance reports. In addition, YouTube has started ranking ISPs based on YouTube streaming performance, which will also hopefully shed a little more light on precisely which Internet bogeymen are to blame for your stuttering video feed.

Filed Under: , , ,
Companies: at&t, comcast, netflix, verizon, youtube

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Netflix, YouTube Streaming Gets Worse on Major ISPs, Who In Turn Blame the Deep, Dark Mysteries of the Internet”

Subscribe: RSS Leave a comment
43 Comments
Anonymous Coward says:

I’d be willing to bet its some kind of routing issue.

I use Verizon and have seen absolutely ludicrous IP routing that often leads to drastically reduced speeds.

Once while transferring a file between my house and a friend about 30 miles across town, I noticed extremely low speeds (we both have 50+ Mbps normally). I tried a trace-route and saw packets being routed through Atlanta, Washington D.C. and Chicago… and we are both in Florida… data was being sent over thousands of miles to travel between two computers in the same city.

TKnarr (profile) says:

This may backfire on the ISPs

Users don’t buy Internet service for the ISP. They buy it for everything else out there. The ISP’s service is just the pipe as far as most people are concerned. If the ISPs degrade service too far, people will start looking for another ISP to get their pipe from.

The major choices for broadband here in San Diego are Cox, Time Warner and AT&T. TW and AT&T ought to worry that, when looking at houses I could possibly buy, one of my criteria is “located in an area serviced by Cox” because I just don’t want to deal with the ongoing headaches I’m sure to have with the other two.

Anonymous Coward says:

Re: This may backfire on the ISPs

You have one maybe two ISPs in most areas and with places like Kentucky pushing legislation to make city/county based ISPs illegal, or extremely difficult to establish or continue on, there really isn’t anywhere else to go. Its like politics, you got a red pile of crap and a blue pile of crap, and no matter the color you have a pile of crap.

Karl Bode (profile) says:

Netflix has commented

Worth adding this investor analyst note that claims Netflix doesn’t believe Verizon is intentionally hampering their streams:

http://recode.net/2014/02/11/netflix-says-verizon-isnt-slowing-down-its-streams/

so far, that?s not happening, Netflix says.

That update comes to us via a note from J.P. Morgan analyst Doug Anmuth, who says he has been talking to Netflix CEO Reed Hastings and CFO David Wells, and they told him they don?t think cable and telco companies are hampering the company?s video streams.

Anmuth doesn?t have much to report on the topic, so here are his comments in their entirety: ?Netflix does not seem overly concerned regarding Net Neutrality, and continues to believe that violations would be escalated quickly. Netflix also indicated that it has no evidence or belief that its service is being throttled.?

Anonymous Coward says:

Re: Re: Re:2 There is a way to tell...

Honestly, I would not be surprised if that is due to routing.

As per my previous post above, bad routing can introduce serious connection slowdown. By going through a VPN, assuming you have a decent connection to it, you can bypass misconfigured or malicious routers ISP side.

Its not entirely unreasonable for an ISP to use bad routing as a way of throttling. That way they can claim to not be “throttling” a connection, and they are technically not… they just route your packets to mars…

Anon77 says:

Re: Re: Re:3 There is a way to tell...

Route all traffic over a peering point that cant handle that much traffic and you have effectively throttled the traffic. Yet you can say it’s not throttled as you haven’t used your throttling specific hardware.

They are being crafty.

virgin media in the UK had terrible youtube streaming for a long time and blamed 3rd party suppliers. These were in fact a local youtube CDN attached to their own network yet they still didn’t give it enough bandwidth to run correctly.

bottom line is never trust an isp, they will screw customers over any way they can

Anonymous Coward says:

Re: Re: Re: There is a way to tell...

Take a close look at the terms of service your ISP gives you. I don’t think it’s been enforced all that often, but some of them don’t allow VPN connections on residential accounts.

So if that becomes a common solution to video streaming, degrading them might not be necessary.

Anonymous Coward says:

Re: There is a way to tell...

That’s incorrect. Let’s say Verizon uses Cogent to get to AWS, and your VPN uses Level3 to get to the VPN point A and it’s bandwidth to get to AWS. If Verizon is jamming the link to Cogent, your VPN could have less latency to AWS and better performance.
The sad fact is, most CDNs like Netflix would gladly peer with Verizon for Free, but it’s in the dual providers best interest not to, ie Verizon and RedBox.
Peering agreements are a lag on the internet, and sadly most network engineers understand this, that is why it’s important to peer with as many providers as possible and keep those links from being saturated.

If you really want to test this out, I would suggest first using mtr with both the vpn and without and seeing where the loss of packets is occuring and high latency points. It’s not full proof but with enough data points and also using looking glass sites, you could narrow down the link with the congestion. Given if you are a residential user not announcing BGP with multiple providers, good luck finding someone to listen.

Anonymous Coward says:

bullshit. the ISPs are deliberately throttling youtube. here in germany it is known fact. Telekom, the biggest provider does deliberately reduce the bandwith because some “issues” with peering agreements, meaning telekom wants to be paid extra for that stuff on top of the regular agreements. Of course that doesn’t fly and the customers have to suffer for it.

so yeah, no mystery here, just plain greed.

Mark Gisleson (profile) says:

More anecdotal evidence

I moved to a remote rural location with no cable or dish options available. I had to set up a Verizon Home Fusion device. It is amazing. It can download the Library of Congress in about half an hour. OK, 30mbps isn’t that fast, but at $5 a GB I had to dump my Netflix account because I couldn’t afford the bandwidth it used.

That and even at 30 mbps, Netflix was choppy and unwatchable. All other streaming video was better, including illegal NBA feeds.

State of the art wireless technology, but it couldn’t stream Netflix worth a crap.

There’s something else going on here.

Anonymous Coward says:

It would seem that much of Netflix’s success was a result of the actions of ISPs’ — by throttling Bittorrent through both active packet-shaping and by extremely restricting the upstream bandwidth.

If restricting Bittorrent traffic was justified by the ISPs for business reasons, why should those same reasons not apply to Netflix, which is a much bigger bandwidth hog?

Mark Milliman (profile) says:

Traffic Equality

The article states the problem very clearly. The problem is that all traffic is treated equal. For better or for worse, all traffic IS NOT EQUAL. Voice is more sensitive than video that is more sensitive than all else. Latency and jitter is increased in port queues that will impact the end performance of a service. By treating all traffic the same your voice and video is bound to be impaired no matter how much bandwidth is thrown at it. As long as you people insist that Net Neutrality means treating all traffic the same, you will be eternally searching for the boogie man.

DannyB (profile) says:

Re: Traffic Equality

Yes, latency matters. For interactive protocols.

But good bandwidth is sufficient for video streaming applications like Netflix, YouTube, etc. It doesn’t matter if your video stream is seconds behind the packets leaving the server, as long as there is bandwidth for a continuous flow of packets to your screen and speakers.

Beech says:

Really,

If Verizon didn’t want people thinking they were violating net neutrality, they shouldn’t have fought a lawsuit over their rights to violate net neutrality. They opened Pandora’s Box, and now every time one of their customers has a hiccup in internet speed they’re going to blame it on throttling. If Verizon really cared so much about “treating all traffic equally” they should have just settled the lawsuit by saying “okay”. Problem solved.

Chronno S. Trigger (profile) says:

The problem with residential connections

This guy states a few facts that suggest a vary possible reason for the discrepancy.

Business connections are dedicated connections, you don’t share the bandwidth. With a residential connection, you do.

Then we can probably safely assume that his business testing was done during business hours and home testing was done after. I have found that during the day my connection to Youtube is amazing, 1080p video, but right around 3:30pm every single day I can’t watch the 144p videos. It picks back up again after 11:30pm. Took me a little while to figure it out, but it turns out that the connection would start sucking right as the kids started getting home from school. Weekends, summer, and the random snow days didn’t have this problem. The kids would come home, play games, watch videos, and kill the internet connection for the entire neighborhood.

The “Deep, dark mysteries of the Internet” are a damn good excuse because they’re true. We’re going to need something much more then random ping tests to prove Verizon is throttling.

Anonymous Coward says:

Latency

A few people have touched on this already, latency is the real killer here not your Mbps.

Q. Whats faster?
A) 100Mbps connection with 1.0 Second of latency
B) 50Mbps connection with 0.5 seconds of latency?

A. Both are the same.

How the data is routed will have a huge impact on latency.
From the data center at work my latency is minimal to nearly everywhere, this is because we are only one hop away from several major providers. (Level3, TW Telecom, XO, Time Warner Cable, AT&T, Hurricane and many more.)

From home on my cable modem the latency on just my ISP’s routers is terrible but oddly the latency to their speed test server is minimal so there is “nothing wrong”

MikeC (profile) says:

Ping of NetFlix.Com — via VPN/Corporate Network vs Via Comcast Direct within 2 minutes of each other

[VPN]

7 59 ms 60 ms 60 ms 206.111.0.65.ptr.us.xo.net [206.111.0.65]
8 135 ms 125 ms 132 ms vb2000d2.rar3.washington-dc.us.xo.net [207.88.13.66]
9 135 ms 141 ms 140 ms te-11-0-0.rar3.sanjose-ca.us.xo.net [207.88.12.69]
10 149 ms 136 ms 131 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226]
11 126 ms 132 ms 129 ms 216.156.84.6.ptr.us.xo.net [216.156.84.6]
12 142 ms 141 ms 138 ms xe-2-2-0-955.jnrt-edge02.prod1.netflix.com [69.53.225.30]
13 140 ms 136 ms 133 ms te1-8.csrt-agg02.prod1.netflix.com [69.53.225.10]
14 136 ms 138 ms 141 ms netflix.co.uk [69.53.236.17]

[DIRECT]

7 28 ms 30 ms 30 ms he-3-2-0-0-cr01.ashburn.va.ibone.comcast.net [68.86.90.177]
8 27 ms 28 ms 26 ms he-0-12-0-0-pe07.ashburn.va.ibone.comcast.net [68.86.83.82]
9 84 ms 32 ms 34 ms dcx2-edge-01.inet.qwest.net [65.120.84.65]
10 * * * Request timed out.
11 104 ms 102 ms 102 ms 63.145.225.58
12 102 ms 103 ms 101 ms te1-8.csrt-agg01.prod1.netflix.com [69.53.225.6]
13 102 ms 102 ms 102 ms netflixinc.com [69.53.236.17]

First 6 entries almost identical just local hops, but when it get’s handed off to the backbone it gets interesting. Note the different destinations depending on the backbone when it get to the same IP address but different domains resolved based on the backbone it rode? Not all that surprising but also when I route local via comcast – look at hop 10 when it’s handed off to Qwest (not surprising in my experience with Qwest networks).

The VPN definitely increases the latency across the board as expected. But the fact that netflix is resolving different FQDN’s to the same IP from different aggregate ports in their network, a touch weird but then I don’t work much with networks with that kind of capacity.

Ever wonder why they call it a CLOUD?? Fully opaque, easy to get lost, only god/buddha/allah know how it works and they aren’t really sure. So as much as I hate big ISP’s I never ascribe malice to what most likely is based in incompetence.

fred says:

maybe

Has anyone considered that perhaps the demand for those services has grown too quickly? Peering is not unlimited, and the amount of total bandwidth cannot increase endlessly, at least not without adding more connections, which costs.

Also, it depends on every step from provider to destination, and the increases in demand now are likely very much like the early days or torrents. Isps cannot add connectivity that quickly to meet a sudden spike in user demand.

Anonymous Coward says:

its not about a “major” isp’s, …..what do “major” anythings usually have?

Alot of customers, …..thats why they are most likely the first to do something not in favour of the people, why would the “pressures that be” pressure a ISP with 500 users, when they want to “affect”, the 2 million market, the bigger a WELL INTENTIONED company gets, the bigger the pressures they recieve, the harder they have to keep their original course………and this is no excuse for them to FAIL this, because they have 2million damn customers, that alone means any bad decisions is MAGNIFIED…….i dont just want to support a top quality service, i want to support a MORAL one, …….you can go on and assume that is my damn fucking line in the sand……..no morals, then piss off with whatever it is your shoveling

Not had me coffee

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »