This article is NOT about net neutrality. I’m hoping that opening declaration serves as a bit of a moat, as from what I’ve seen in the past few weeks any point of view on net neutrality is met with serious vitriol. I don’t need contempt in general, especially over something that I do for free. So while I stand by my many requests for comments, head elsewhere if you’re going to flame me.
If you are reading this article, odds are extremely good that you are on the Internet (given that EE Journal does not offer fax or mail service). And if you are on the Internet, odds are extremely good you’ve been reading about the Netflix-Comcast feud. Which from my perspective—and I am not being sarcastic at this point—is NOT about net neutrality.
In the interest of full disclosure, I am doubly biased: I am both a Netflix customer AND a Comcast customer. I’m not certain of the Socratic logic, but I’m going with “being biased on both sides of the feud cancels out” and leaves me unbiased.
I am not going to review the entire history of the Netflix-Comcast feud, which would give me a splitting headache. In some sense, I am writing this article BECAUSE the whole thing has given me a headache and this is my way of getting it out of my system. I apologize if you’re in the same boat, because now I am further contributing to your headache. Get it out of your system by adding a constructive comment below; you’ll feel better.
I’ll start by admitting—as I often do in the interest of transparency—that I am no expert on net neutrality. My simple definition: on any given public segment of the Internet infrastructure, all packets are handled without preference. Favorites are not permitted vis-à-vis providers, for example, all VoIP traffic flows the same regardless of vendor. Favorites are not permitted vis-à-vis packet payload, so instant messaging packets, email packets, streaming audio packets, and streaming video packets all flow across the Internet and to our homes in the same manner.
Astute observers may note that handling different packet payloads in a different manner could be quite desirable: audio and video traffic are quite susceptible to indeterminate latency, whereas rather few people care if their email shows up five seconds late. Indeed, this ‘discrimination’ is called quality of service (QoS), and entire companies exist on the basis of their QoS algorithms that optimize traffic across corporate networks. Heck, I’ve configured QoS on my home network: video traffic (in my case, Netflix, Amazon and Skype) has priority over audio; audio traffic (in my case Sonos and Amazon) has priority over all the other bits. Digression aside: Comcast and other major broadband ISPs are NOT performing QoS today, though they do make noises about doing so from time to time.
Many months back, I noticed that Netflix was freezing with increasing frequency; it seemed that my Comcast broadband connection was not providing enough bandwidth to keep the Netflix player buffer filled. Quite frustrating, given that I am paying BOTH Netflix and Comcast so that I can watch video without pauses that I cannot help but liken to commercial breaks (albeit short ones). Whenever a particular video froze more than twice an hour, I thought of Alan Shepard barking, “Fix your little problem and LIGHT this candle!”
So when the Netflix-Comcast feud first broke, I thought “Reed Hastings is bent out of shape because Comcast has set up QoS to favor their own video-on-demand over Netflix.” It would be fairly straightforward to perform Comcast-Netflix tests across America, streaming “The Right Stuff” a few thousand times and measuring the results. I assumed Reed had such proof in some spiffy charts illustrating the blatant favoritism and maybe a Photoshopped picture of Brian Roberts flipping off the FCC.
Turns out, Reed had nothing of the sort. Because Comcast was doing nothing of the sort.
An important digression is in order here, to explain one particular feature of the Internet infrastructure: the content delivery network (CDN). As we all learned from Senator Ted Stevens in 2006, “The Internet is not a big truck. It’s a series of tubes.” Ignoring for a moment the infamy Senator Stevens earned with that declaration, taken as an analogy it serves my purpose. Except in my analogy, the Internet is both a series of tubes AND a big truck.
One of the significant issues the internet had in 2006, and still has today, is the number of hops required to get from the (website, audio, or video) server to you. If we use the pneumatic tubes analogy—think of an Internet documentary directed by Terry Gilliam—you need routers scattered across the world that switch your container from one tube to the next across a vast network crisscrossing America. In reality the Internet consists of, well, routers scattered across the world that, well, switch your packets from one (fiber optic or copper) trunk to the next across the vast network crisscrossing America. Each of those hops adds latency; and getting from the server to you often requires LOTS of hops.
Enter the CDN. Companies like Akamai built their own networks that transport data from a server to a “cache” (I’ll get to the quotation marks in a second) physically located very close to—ideally collocated with—network operation centers near major population centers. The beauty of a CDN is two-fold: [a] there are VERY few hops across their high-speed networks from a server to the caches and [b] there are just a FEW hops from their caches to you. Back to what I called the CDN caches for a moment: while they do “cache” websites in the computer architecture sense of the term, they are network-attached storage for audio and video data. Think of trucks filled with audio and video data making a beeline from the servers to the caches; mission accomplished: Senator Stevens, rest in peace.
Heavily trafficked providers—the world’s busiest websites and audio and video streaming services—would experience massive bottlenecks without CDNs. The local CDN caches effectively replicate the aforementioned providers and place the copies very close to the consumers. What a great service the CDNs provide! Kudos to them for their charity and generosity! No, no, just kidding: of course Akamai and the other CDNs charge data providers for content distribution. And for massive data providers, the CDNs charge serious money.
Now I am sure you’ve heard that Netflix is the definitive massive data provider, generating one-third of all Internet traffic in North America. That is an impressive stat, all the more so when you realize that Twitter generates only one-quarter of all Internet traffic in North America. No, no, I’m kidding: Twitter just SEEMS to generate one-quarter of all Internet traffic in North America. (Favor to ask: would someone please teach “The First Rule of Holes” to athletes and celebrities on Twitter?)
Clearly Netflix MUST utilize the mother of all CDNs, otherwise the Internet’s tubes would get royally and truly clogged (think Terry Gilliam again). So Netflix pays Akamai a ton, right? Nope, not a dime. Reed Hastings is an exceptionally smart dude with tremendous vision, so over the past few years Netflix built their own mother of all CDNs. This sounds like the END of our story—gargantuan video libraries scattered across North America—but it is really the BEGINNING of our story.
“OK then,” you’re thinking, “Netlfix has their massive video library replicated N times across the United States, with local copies located near all major population centers. Netflix is home free.” Those quaint last four words are where you would be dreadfully wrong. Though you did pick well: the entire Netflix-Comcast feud comes down to just three words: Netflix, home, free.
Netflix believe that broadband providers like Comcast should deliver videos from the Netflix CDN to your home for free. Netflix, home, free; CHECK.
Comcast believes that Netflix has lost its marbles if they think that gaining unfettered access and hoovering up fully one-third of their broadband network delivering videos to your home should be free. Netflix, home, free; CHECK.
Reed Hastings is feverishly screaming that all packets are created equal. The Netflix CDN brings their video packets to the metaphorical doorstep of the Comcast broadband network, and it is Comcast’s responsibility to deliver those packets to your home. Rain, sleet, snow, one-third of their bandwidth be damned. Netflix dropped serious coin building that CDN and they should NOT have to pay a penny to Comcast or any other ISP to get all those video packets delivered to your house.
You will no doubt be shocked to learn that Comcast doesn’t see it that way. Despite being the second-lowest rated North American company on customer satisfaction, Comcast has built an impressive broadband network. And it cost a mint to build that capacity: those umpteen thousands of miles of coax they laid down were never intended to move bits, it was intended to move analog … in ONE direction, TO your house. They’ve re-tooled that cable plant—the little boxes on your street, the bigger boxes in your neighborhood, their distribution centers, pretty much everything EXCEPT the coax—to handle bi-directional digital traffic and deliver up to 100 Mbps Internet PLUS a ton of digital video to your house.
Modest historical digression. Those of you with really good memories may recall that the very first cable modems used analog return. Yup, upstream traffic was sent via analog modem, as cable plant of the time was incapable of sending data in the ‘wrong’ direction.
Having spent all that money building a solid broadband network, Brian Roberts wakes up one day and discovers that one-third of his traffic comes from one source: Netflix. If he were charitable and generous type, Brian would write another giant check, increase the bandwidth available on his backbone and all of us would thank Brian for being so devoted to net neutrality. Guess what? Your Netflix would still freeze a couple of times an hour. Because the problem is NOT a shortage of bandwidth on the Comcast broadband network.
“Hold on there dude,” you’re wondering, “if Comcast has enough bandwidth and my video still stalls, then they must be de-prioritizing Netflix traffic. So this IS all about net neutrality after all!” I stated clearly that this article is NOT about net neutrality, so obviously I am not about to pull a 180. However, it is perfectly fair to ask: “If Comcast has the bandwidth and all packets flow in the same manner—but our videos still freeze—what exactly is the problem?”
The problem is in the pipes; well, to be more precise, the problem is the connection BETWEEN the pipes. As noted above: Netflix built the mother of all CDNs, effectively constructing REALLY fat pipes to all major population centers; Comcast built a broadband network that impresses ME (someone not easily impressed by ISPs) with its own set of fat pipes. The entire issue is the CONNECTION between the fabulously capable pipes owned by Netflix and the tremendously impressive pipes owned by Comcast.
Think of it this way: Netflix drops a massive video library server in your town. How does that Netflix server route those video packets to your home? Over a public internet backbone and then on to Comcast’s broadband network? Absolutely NOT: that would add hops, which you will recall is every bit as bad as crossing the streams. To say nothing of the fact that the public internet backbone would come to a screeching halt under the weight of all that traffic. This is why we invented CDNs in the first place.
The solution is the answer to the age-old question “what is the shortest distance between two points?” No, no, “Let me check Google Maps” is most certainly not the answer. The shortest distance is on a straight line: connect the Netflix CDN DIRECTLY to the Comcast backbone. As Inspector Clouseau would say, “Voilà! The problem, she is solve-ed!” Believe it or not, Netflix and Comcast agree on this, which is good, because it is the completely logical solution.
But…not so good, because Netflix feels quite strongly that they should get this direct connection at no charge, while Comcast—wacky capitalists that they are—think that such a high-value-high-bandwidth connection has, well, value that Netflix should pay for. And THAT is the source of the feud.
Lest you think any of this would be easy to explain, it gets harder to explain. Netflix convinced many smaller ISPs that a direct connection to their backbones was good for the ISP’s customers and hence should be provided to Netflix free of charge. The larger ISPs like Comcast had the (pardon the pun, but I simply must) backbone to tell Netflix “you want the high-value-high-bandwidth connection, you’re going to pay for it.” And travel with me now through the looking glass: Netflix agreed and paid Comcast.
Inspector Clouseau nailed it: as soon the Netflix CDN was connected directly to the Comcast backbone, customer complaints about stalled/frozen videos plummeted.
I’ll refrain from quoting you and what you’re thinking, but I would actually like to know what you’re thinking at the end of this paragraph. It was AFTER he wrote the check to Comcast that Reed Hastings went on the bender of all Twitter-and-then-some whistle stop tours telling everyone that Comcast was spitting on net neutrality, destroying the Internet and—while they lost track of the photo—that Brian Roberts flipped off the FCC. Seriously, I can’t make up stuff like this even when I focus.
I will oh-so-lightly touch on the well-established business of “unbalanced peering,” as there is standing precedent here. When traffic flows from one Internet backbone to another with roughly equal bits exchanged in each direction, both providers are copacetic and nobody owes any money. When traffic flow is asymmetrical, however, peering agreements call for the provider pushing more bits than it receives to pay the other party for the imbalance. Given that CDNs were invented to be nearly unidirectional fire hoses, the standard arrangement calls for the CDN to pay the Internet provider on the receiving end of the fire hose. Hmm, that sounds strangely familiar …
One last analogy and then we’ll hand this to the jury. Gross oversimplification here, but the analogy works:
- Imagine that there is a single router sitting between the public Internet and the Comcast broadband network
- The Netflix CDN is connected directly to the Comcast backbone
- Therefore, Netflix video packets no longer ‘compete’ with all other packets at that router; they do an end-run around the potential bottleneck
- Now that Netflix packets have nearly unfettered access to the Comcast backbone, they hoover up all the bandwidth they need; THAT IS HOW AND WHY videos no longer freeze
- Given that nobody added bandwidth to the Comcast broadband network, other traffic MUST be slower than it was pre-Netflix-Comcast-direct-connection
Call me cynical, but THAT doesn’t sound like a shining model of net neutrality. That sounds like Netflix packets are getting preferential treatment at the expense of other packets. That sounds like Reed Hastings should put a cork in the “Comcast is undermining net neutrality” bandwagon.
About the Author:
Bruce Kleinman is a senior technology/business executive and principal at FSVadvisors and blogs on fromsiliconvalley.com
The problem is that streaming video in almost all forms, are UDP packets which do not respond to flow control well. Nearly all other packets are TCP, and reduce window size, and real bandwidth demands, using packet loss as a flow control.
I operate a small (150 family rural mountains WISP). The FCC says that I can not selectively throttle high bandwidth streaming services, which are used by 20% of my customers, and that the other 80% must accept the extremely poor service that results from streaming UDP and network saturating P2P services.
Franky, the other 80% are paying the majority of the costs, and I have to question why they must accept inferrior service while the streaming UDP and P2P users hog ALL the network bandwidth.
As far as Netflix, they refuse to provide a means for small regional ISP operations to have a local CDN or to cache their content …. so it does staturate not only the expensive internet feed but also saturates last mile network resources, like Access Points for a WISP. The Netflix CDN program only benefits large ISP operations, which are near a major backbone peering point, with very cheap internet access …. 95% of the small ISP operations do not have access to cheap peering bandwidth.
I could support network neutrality, if all the services were equally polite and willing the share the network resources fairly. Frankly, baning all UDP applications over the network would be a big start …. and allowing the service provider to throttle P2P applications running dozens of parallel streams to hog bandwidth was legal.
Totally, There are a number of strategies that you can use. I come from the CDN and Telco world and the direct connect voila it works is really due to Preffing more than anything. Looking at the source of the traffic, you can assign unusually high latency details in your BGP routing table that make Netflix or other services wait or enter over less expensive peers or transit connections. This is not a violation of NetNeutrality since NN only deals with Priority after the packet is on your network – however there are no statutes stating you must allow the packet through the most efficient gateway.
Netflix should be buying bandwidth from tier 1 providers and eyeball networks. There is no reason why they should be able to buy super cheap bandwidth and then have cart blanche to any network in the world. Comcast, AT&T, Verizon, and others have been de-peering non-Tier1 networks over the past few years to ensure the leeches need to buy bandwidth at a reasonable cost in order reach valuable end users, which is the reason for Akamai’s deal with AT&T 2 years ago and why Verizon picked up Edgecast last year. This is a trend that is going to continue – Google, Apple, Microsoft, and other companies with a business model pay for the bandwidth that they use, so why shouldn’t Reid? Call him smart, call him shrewd, but he is the one who has caused more damage to the Net Neutrality discussion because his spin circles around his unwilling to pay for what he uses – its like buying a “hamburger steak” in a restaurant and upon receiving it, you make the argument that because you ordered “steak” a fillet you have been served.
The real issue at point here is, if Netflix had to actually pay for the bandwidth they used, would they have a sustainable business model? Could they sustain their own growth if they were responsible for the whole bill and the answer is no. Their business model requires that EVERY network subscriber in the world underwrite some portion of their consumption in order for their business to be valid – in short, $10.99/month doesn’t put food on the table and the lights on…
jroza1, most small ISP operations only have a single provider, so we do not have the ability to externally force streaming traffic to one provider/circuit, and all other traffic to another that is more lightly loaded. When 8 bonded T1 connections from Level3 cost $2,600/mo, it’s not like a lot of small ISP operations have a lot of room to just magically ask for a free 100mbps connection to serve Netflix.
I believe it’s the end users that should be paying for the high bandwidth …. except that in this market, pure metered internet isn’t well received. I’d loose about 3/4 my customers to other options, like satellite.
In the end, network neutrality is going to put a lot of small providers out of business, simply because the vocal Network Neutrality people, are also the ones wanting a free lunch.
The other victim, is VoIP users, that have their voice quality completely drop out in the evening with streaming video choking the pipes into saturation.
If both streaming video and VoIP were TCP, it would be a LOT less of a problem.