Segnalo il seguente articolo, in tema con il de.peering di Telecom Italia -- Luca http://arstechnica.com/information-technology/2013/07/why-youtube-buffers-th... Why YouTube buffers: The secret deals that make—and break—online video Aurich Lawson Lee Hutchinson has a problem. My fellow Ars writer is a man who loves to watch YouTube videos—mostly space rocket launches and gun demonstrations, I assume—but he never knows when his home Internet service will let him do so. "For at least the past year, I've suffered from ridiculously awful YouTube speeds," Hutchinson tells me. "Ads load quickly—there's never anything wrong with the ads!—but during peak times, HD videos have been almost universally unwatchable. I've found myself having to reduce the quality down to 480p and sometimes even down to 240p to watch things without buffering. More recently, videos would start to play and buffer without issue, then simply stop buffering at some point between a third and two-thirds in. When the playhead hit the end of the buffer—which might be at 1:30 of a six-minute video—the video would hang for several seconds, then simply end. The video's total time would change from six minutes to 1:30 minutes and I'd be presented with the standard 'related videos' view that you see when a video is over." Hutchinson, a Houston resident who pays Comcast for 16Mbps business-class cable, is far from alone. As one Ars reader recently complained, "YouTube is almost unusable on my [Verizon] FiOS connection during peak hours." Another reader responded, "To be fair, it's unusable with almost any ISP." Hutchinson's YouTube playback has actually gotten better in recent weeks. But complaints about streaming video services—notably YouTube and Netflix—are repeated again and again in articles and support forums across the Internet. Why does online video have such problems? People may assume there are perfectly innocent causes related to their computers or to the mysterious workings of the Internet. Often, they're correct. But cynical types who suspect their Internet Service Providers (ISPs) intentionally degrade streaming video may be right as well. No, your ISP (probably) isn't sniffing your traffic every time you click a YouTube or Netflix link, ready to throttle your bandwidth. But behind the scenes, in negotiations that almost never become public, the world's biggest Internet providers and video services argue over how much one network should pay to connect to another. When these negotiations fail, users suffer. In other words, bad video performance is often caused not just by technology problems but also by business decisions made by the companies that control the Internet. Please wait. Keep waiting. These business decisions involve "peering" agreements that Internet companies make to pass traffic from one to another and negotiations over caching services that store videos closer to people's homes so they can load faster in your browser. When Internet providers refuse to upgrade peering connections, traffic gets congested. When ISPs refuse to use the caching services offered by the likes of Google and Netflix, video has to travel farther across the Internet to get to its final destination—your living room. The negotiations can lead to brinksmanship and bad blood. Recent public examples of such spats include: November 2010: After Internet backbone provider Level 3 signs a deal with Netflix to distribute video, Comcast demands money from Level 3 for carrying traffic over the proverbial "last mile" to Comcast subscribers. January 2011: European ISPs Deutsche Telekom, Orange (formerly France Telecom), Telecom Italia, and Telefónica commission a report saying companies like Netflix and Google's YouTube service should give ISPs a lot more money. August 2011: Cogent, another Internet backbone provider that handles Netflix traffic, files a complaint in France against Orange, saying the ISP is providing inadequate connection speeds. January 2013: Free, a French ISP, is accused of slowing down YouTube traffic by failing to upgrade infrastructure (but is later cleared of intentionally degrading YouTube traffic by the French regulator). Free also temporarily blocks ads on YouTube and other video services by sending an update to its modems. January 2013: Orange and Google have a similar dispute, with Orange CEO Stephane Richard claiming victory. He says that Google is paying Orange to compensate the operator for mobile traffic sent from Google servers. January 2013: Time Warner refuses Netflix's offer of a free caching service that would provide better performance to Netflix users on Time Warner's network. June 2013: Cogent accuses Verizon of allowing "ports" between the two providers to fill up, degrading Netflix performance for Verizon customers. July 2013: The European Commission opens an antitrust probe into whether ISPs abused market positions in negotiations with content providers, and it searches the offices of Orange, Deutsche Telekom, and Telefónica. Separately, the French government demands details of interconnection agreements involving AT&T and Verizon. In the most extreme cases, large Internet companies stop passing traffic to one another entirely. (This happened in 2005 with France Telecom and Cogent, in 2005 with Cogent and Level 3, and in 2008 with Sprint and Cogent.) But recent disputes have been less likely to lead to a complete severing of ties. "That type of reaction to a policy is becoming less common, possibly because it's so easy to publicize it," Reggie Forster, director of network engineering at XO Communications, told Ars. "They tend to want to keep that quiet." Instead, network operators can degrade traffic by failing to upgrade connections without severing them entirely. The public won't realize that's what's going on unless negotiations become so contentious that one party makes them public—or a government decides to investigate. Degraded connections disproportionately affect the quality of streaming video because video requires far more bits than most other types of traffic. Netflix and YouTube alone account for nearly half of all Internet traffic to homes in North America during peak hours, according to research by SandVine. And customers are far more likely to be annoyed by a video that stutters and stops than by a webpage taking a few extra seconds to load. Enlarge / North American traffic during peak hours. SandVine To get to the root of these problems, we need to take a step back and talk about the Internet itself. The very name "Internet" suggests many networks interconnecting, but few people know how the terms of these connections are negotiated. Understanding the business relationships that allow the Internet to exist in its present form is crucial to understanding the subtle and not-so-subtle ways Internet companies can bring your YouTube and Netflix videos to a slow stutter. Peering—the backbone of the Internet The core of the Internet, the closest thing it has to a "backbone," is a dozen or so networks consisting of data centers throughout the world. These networks, operated by private businesses, are called "Tier 1" because they can reach every part of the Internet simply by peering with one another. Two Tier 1 networks peer with each by configuring routers at multiple data centers to pass traffic back and forth. Peering is a point-to-point connection only, which doesn't necessarily guarantee passage of traffic to any networks beyond the two involved in the deal. "Both providers have a large router, a Cisco or Juniper router, and they agree to interconnect them," Mark Taylor, VP of media and IP services at Level 3, told Ars. For example, he said, "A 10Gbps port on one is connected to a 10Gbps port on another. The only traffic that can flow between those connections are between the respective customers of both parties." Tier 1 networks don't need to buy "transit"—an arrangement where one company pays another to accept its traffic and distribute it to all networks connected to the Internet. Smaller networks do, although even companies as big as Comcast buy transit. Peering and transit are essential to Netflix and YouTube traffic. The Internet backbone providers that Netflix and Google pay to carry their traffic to the rest of the Internet need robust connections to other networks to ensure delivery of streaming videos to customers of all ISPs. Netflix and Google also offer to peer directly with ISPs to give videos a more direct path to customers' homes. (For more discussion of peering and transit, see Peter Bright's article "Can a DDoS break the Internet? Sure… just not all of it.") Defining which networks are "Tier 1" isn't as simple as you might expect. The volunteer editors at Wikipedia have listed Tier 1 networks as AT&T, CenturyLink, Deutsche Telekom, XO Communications, Telecom Italia, GTT, Verizon, Sprint, TeliaSonera International Carrier, NTT, Level 3, Tata Communications, and Zayo Group. The list is probably not complete. Wikipedia calls Cogent and Orange "major networks" but not Tier 1s, meaning they don't have to buy transit but do pay for peering. (Cogent denies this, telling Ars that it pays no other provider for interconnectivity.) Whether payment should disqualify a network from being labeled "Tier 1" is under dispute. A packet is routed across the Internet exactly the same way whether it is routed as the result of a free peering agreement or whether it is routed as part of a paid peering agreement, notes Martin Levy, director of IPv6 strategy at IP transit provider Hurricane Electric. Hurricane says it is a Tier 1 network for IPv6 traffic, though not for IPv4 traffic. The reason financial agreements are important isn't so much the semantic question of who is a Tier 1 network; it's that disagreements over whether payments should be made end up harming the quality of end-user connections. "Everybody always wants to get paid," Rudolf van der Berg, an economist and policy analyst for OECD, told Ars. "If I had the choice between paying you, peering with you, or getting paid by you, I would choose getting paid by you. It doesn't necessarily mean the other party is interested in paying. And that's the problem here." Peering without payment—"settlement-free" peering—occurs quite often between networks of roughly equal size and negotiating power. "The word peering generally is synonymous with settlement-free," Taylor said. "There is an agreement by both parties that there is an equal benefit by each to connect to the two networks." Typical peering agreements between US-based Tier 1 networks involve connections in six to 10 locations across the US, mainly in major cities, Forster said. XO also peers with providers in London, Amsterdam, and Frankfurt. A crucial part of peering agreements is upgrading connections when they start to fill, because congested ports slow down the rate at which bits flow between the networks. That's what happened when Verizon refused to add as many ports to its connections with Cogent as Cogent felt was necessary. Verizon FiOS Customers promptly complained on Verizon support forums when their Netflix video playback got dramatically worse. Netflix speed by carrier. Netflix "Typically what happened is when the connections reached about 50 percent utilization, the two parties agreed to upgrade them and they would be upgraded in a timely manner," Cogent CEO Dave Schaeffer told Ars. "Over the past year or so, as we have continued to pick up Netflix traffic, Verizon has continuously slowed down the rate of upgrading those connections, allowing the interconnections to become totally saturated and therefore degrading the quality of throughput." Schaeffer said this is true of all the big players to varying degrees, naming Comcast, Time Warner, CenturyLink, and AT&T. Out of those, he said that "AT&T is the best behaved of the bunch." Letting ports fill up can be a negotiating tactic. Verizon and Cogent each have to spend about $10,000 for equipment when a port is added, Schaeffer said—pocket change for companies of this size. But instead of the companies sharing equal costs, Verizon wants Cogent to pay because more traffic is flowing from Cogent to Verizon than vice versa. "When the traffic loads are not symmetric, the provider with the heavier load typically pays the other for transit," Verizon said in a blog post. Cogent doesn't want to pay and says the traffic loads are unequal because Verizon customers are requesting so many Netflix videos. Negotiations are surely ongoing behind the scenes, but no agreement has been announced. The dispute is similar to the one that happened in late 2010, when Comcast demanded payment from Level 3 in order to carry Netflix traffic. The dispute didn't get to the point where ports were saturated. "We, under protest, agreed to pay," Taylor told Ars. This month, nearly three years after the dispute began, the companies released a joint statement. "Level 3 and Comcast have resolved their prior interconnect dispute on mutually satisfactory terms. Details will not be released," it said. "The funny thing" about these disputes is how little money is involved, van der Berg said. The French telecom regulator ARCEP has found that money changing hands between operators for peering and transit, plus the amount paid to third parties that host Internet exchange points, is equal to just 0.2 percent to 0.5 percent of revenue generated by the supply of Internet access to end users. One obvious question to ask is whether ISPs refusing to upgrade peering connections could violate net neutrality laws. The Federal Communications Commission's (FCC) Open Internet Order prevents ISPs from blocking content or unreasonably discriminating against third-party traffic. ISPs that sell their own video services while simultaneously degrading other video traffic might seem to be violating the principles of net neutrality, but they're likely not violating the letter of the law. The FCC's Open Internet Order makes only one reference to peering: "We do not intend our rules to affect existing arrangements for network interconnection, including existing paid peering arrangements." Susan Crawford, a former tech policy advisor to President Obama and author of Captive Audience: The Telecom Industry & Monopoly Power in the New Gilded Age, told Ars via e-mail that this exception for network interconnection agreements including paid peering "is an enormous problem. If we focus only on discrimination vis a vis users' access to applications, we're missing a huge part of the story. Network operators in the US are now sufficiently powerful that they can pinch traffic farther upstream as well as pinch traffic in the last mile." It's easy to see why more traffic flows from Cogent to Verizon than vice versa. Cogent is one of Netflix's Internet service providers—Netflix pays Cogent for transit, to carry its traffic to the rest of the Internet. Many Verizon customers, meanwhile, pay Netflix to stream video. Each time a Netflix customer using Verizon's network clicks on a Netflix video, that click sends just a tiny bit of traffic to Verizon and then on to Cogent and Netflix. In return, that customer receives a flood of data—up to 2.8GB for each hour of high-definition content. This is just how the last-mile ISPs want it, apparently, since they have made it difficult for customers to send as much traffic as they receive by providing far lower upload speeds than download speeds. Even customers who might try to achieve some upload/download parity—perhaps by using backup services to replicate all of their data—will find that hard to do. Trying to get paid on both ends Everyone involved in sending Netflix traffic to customers is getting compensated. Verizon and Netflix get paid by home Internet users, and Cogent is paid by Netflix. Verizon wants to get paid twice—by its own Internet customers and by Cogent. The question of whether Cogent should pay Verizon comes down in part to the question of who causes the traffic. Did Netflix "cause" the traffic by offering streaming content over the Internet? Or did Verizon cause the traffic by providing Internet service to customers who expect access to Netflix videos? "Verizon has chosen to sell its customers a product that they hope those customers don't actually use," Schaeffer said. "And when customers use it and request movies, they have not ensured there is adequate connectivity to get that video content back to their customers." Schaeffer accused Verizon of favoring its own services. "Verizon is trying to incent customers to buy its video product, its Redbox product, as opposed to using competitive products of Netflix and others," he said. Verizon might turn Schaeffer's arguments on their head, saying Netflix is selling a streaming video service without ensuring that the videos have a clear path to all customers. Verizon declined repeated interview requests from Ars, however, as did the majority of companies involved in these disputes. Comcast, Time Warner Cable, and CenturyLink did not answer questions or make anyone available for interviews. Google and Netflix both declined to say anything on the record. AT&T did not respond to e-mails. Three European players involved in disputes—Free, Orange, Deutsche Telekom—did not reply to e-mails. Telefónica responded that "we can’t comment on regulatory cases that are ongoing except to say that Telefónica is cooperating fully." The FCC did not make anyone available for an interview or answer our questions. Since ISPs have argued that YouTube, Netflix, and other video services should pay for access to customers, they're actually seeking payment from three sources: their own customers, the online video providers, and Tier 1 networks like Cogent and Level 3. When asked if ISPs are degrading Netflix and YouTube traffic to steer users toward their own video services, Crawford told Ars that "the very powerful eyeball networks in the US (and particularly Comcast and Time Warner Cable) have ample incentive and ability to protect the IP services in which they have economic interests. Their real goal, however, is simpler and richer. They have enormous incentives to build a moat around their high-speed data networks and charge for entry because data is a very high-margin (north of 95 percent for the cable companies), addictive, utility product over which they have local monopoly control. They have told Wall Street they will do this. Yes, charging for entry serves the same purposes as discrimination in favor of their own VOD [video-on-demand], but it is a richer and blunter proposition for them." Speed problems in both the US and Europe In addition to the Netflix complaints by Verizon customers, Time Warner Cable customers recently complained about poor YouTube service. One reddit thread shows how some technically inclined Time Warner customers used traffic analysis tools to identify the cause of YouTube problems. They were able to improve performance by blocking certain IP addresses, likely ones associated with slow YouTube content servers. Time Warner Communications Director Jeff Simmermon wrote in a blog post last month that a "small subset of our customers... seem to think we are intentionally degrading their service. If I were to sum up these kinds of complaints, it would go something like this: 'Hulu and YouTube wouldn’t intentionally degrade their services and provide a subpar experience, but for some reason, it makes total sense that Time Warner Cable would by throttling customers’ traffic.'" Simmermon further noted that "some, but not all, online video providers have the resources to store copies, or caches, of their videos on servers that are a part of a Content Delivery Network, or CDN. When a user clicks on a link to a particular video, the Internet provider can quickly determine where the nearest cached copy of the video is on the CDN and deliver it. More popular videos will have more cached copies, with better performance, while less popular videos may be stored in fewer, further places." What he didn't comment on is whether Time Warner accepts Google's YouTube caches into its network. But as noted earlier, Time Warner has refused to accept Netflix's caching equipment. Netflix may still be hosting its own caches throughout the country, but if they're not in Time Warner data centers, Time Warner customers get worse performance than they otherwise might. Europeans are suffering similar problems. "There is a kind of hot/cold thing," said telecom watcher Benoît Felten, the CEO of Diffraction Analysis. Felten lives in France and is a customer of Free, an ISP that has been accused of throttling YouTube, or at least under-investing in the connections that pass YouTube traffic to end users. "Right now, YouTube doesn't work too bad on Free," Felten said earlier this month. "Three weeks ago it was horrendous. A 30-second video, like an Angry Birds solution video, would take 12 minutes before the first frame moved. Right now, you get some lags but it's acceptable. I suspect whatever they're doing, they're constantly shifting it so it doesn't look like it's constantly horrible." Instead of intentional discrimination against YouTube, it may simply be that Free's network is short of capacity. A few days after Felten spoke with Ars, the French telecom regulator ARCEP announced that it found no evidence of Free using discriminatory practices against YouTube, although it did say traffic is congested during peak hours. The European market has far more competition for home Internet service than the US does. Felten says he is going to switch to a different Internet service and hopes many other Free customers will do the same. While European ISPs want payments from Google and Netflix, Felten said they should just focus on improving Internet service. "Customers are already paying for it," he said. "You sell a service to the end user which is you can access the Internet. You make a huge margin on that. Why should they get extra revenue for something that's already being paid for?" Paid peering contracts with fixed prices also don't make sense, he said, because IP transit is "constantly going down in price... At some point you'll be paying more for peering than you would for transit, which is absurd." To illustrate the absurdity of ISP proposals to charge video providers, Felten noted that "when Orange purchased [video service] Dailymotion, that same week a white paper released by A.T. Kearney that Orange had financed basically said, 'this is how much online service providers should pay per gigabit of traffic that is delivered through our networks.' I applied that calculation to Dailymotion's traffic, and basically they would have had to pay twice their annual revenue in charges (nearly €42 million [roughly $55.5 million]). It's absurd. The whole economic model would break down if you were to apply that." New, improved way of measuring traffic Concerned about ISPs demanding payment from fellow network peers, Level 3 has come up with a new way of measuring peers' impact on each other's networks. Traditionally, traffic loads have been thought to be "in balance" if each peer sends about as much traffic to the other peer as it receives. But the direction in which traffic flows has no impact on how much it costs to carry it, Level 3 argues. Because video streaming traffic dominates the Web, so-called "eyeball" networks (ISPs who deliver traffic over the last mile) can never be in balance with the networks that deliver video under the old measurement. Instead, Level 3 wants to measure via "bit miles," the distance traffic is carried and the number of bits carried, regardless of which direction the traffic flows. So far, tw telecom and XO Communications have entered long-term, settlement-free peering agreements with Level 3 using bit mile measurements. "This is a new idea. I think it's going to take a little bit to get more people on board," Forster said. "It needs to be simplified. Right now, both us and Level 3 are working out kinks in how this works." Two networks could still end up with unbalanced traffic under the bit miles approach, but it would be easier to fix. "If you connect in 10, 15, 20, 30 places, it's easy to add connectivity points, it's easy over time to add and subtract those connectivity points to make sure the networks are in balance and you incur equal costs of transmission," Taylor said. Level 3 objects to most demands that peers pay each other, even if the amounts aren't huge. The amount of money "for Level 3 is not material," Taylor said. "It's the principle of who decides because it's a non-competitive market. We have no choice. There is one party that sets the rate; there is no competitive pressure we can bring to bear to negotiate that rate or get a different way into the network." This is where a lack of competition in the home Internet market comes into play. When a consumer's second option is slow or non-existent, the dominant Internet provider can maintain a monopoly even while delivering poor service. According to FCC data, 86 percent of American households have a choice of at least two wired Internet providers offering 3Mbps download speed and 768Kbps upload—but that's not even fast enough to qualify as broadband. Only 34 percent of US households have a choice of Internet providers offering at least 6Mbps down and 1.5Mbps up. The FCC defines broadband as 4Mbps down and 1Mbps up. Netflix recommends 5Mbps for HD streams—and remember that a home Internet connection is often shared across multiple devices and people. Moreover, speeds promised by ISPs may guarantee a certain level of bandwidth between you and your ISP's network, but not between you and the rest of the Internet because of peering disputes and under-provisioned connections. Enlarge / From the FCC report "Internet Access Services: Status as of December 31, 2011," released in February 2013. FCC The argument that payments between network operators are required because streaming video throws traffic loads out of balance "is a complete canard," Crawford said. She explains: When the commercial Internet was born, the telcos exchanged traffic with each other using handshake peering arrangements for which the paradigm was "ratio-based" payments. Because phone traffic was basically 1:1, or often 1:1, upstream and downstream, the big actors peered with one another for free. Today, we have enormously powerful eyeball network operators who are still taking advantage of this peering paradigm. And so they say, "my users are asking for more traffic than they're sending upstream." Using this rationale, the cable companies have moved from having to pay to get their users' traffic out onto the Internet (which is where they were when the Internet began) to demanding free peering, to, today, charging for interconnection. But the use of the network, and the architecture of the network, is now completely different. Cable is asymmetric; it was built for passive downloads and does not favor uploads. Also, users are doing a lot of downloading of content. It's a user-driven Internet. So of course traffic volumes are greater downstream than upstream. That's what cable is selling. It should want to sell more; it should want to build capacity and connect freely with transit networks; and it would if there was any competition. But there isn't. So cable companies can draw up the drawbridge and charge with impunity, or refuse to add capacity to service interconnecting providers unless they're paid (which is the same thing). John Bergmayer, senior staff attorney at advocacy group Public Knowledge, said ISPs like Verizon can't credibly claim that their networks are overloaded by Netflix. "We've seen all the cable advocates and the NCTA (National Cable & Telecommunications Association) back away from the claim that data caps are necessary to manage congestion," Bergmayer said. "Networks really aren't saturated on the last mile like they have been trying to claim for years. Particularly with Verizon—with FiOS, it has huge capacity. Verizon has said again and again that it could upgrade to gigabit; it just doesn't think that consumers really want it. They're always bragging about how great their fiber network is. They can't have their cake and eat it, too, and not really have real capacity problems on the one hand except when it comes to hammering out these deals with the Internet content companies on the other hand." While European authorities have been demanding data on peering agreements from Internet companies, Bergmayer said the FCC should do the same. "Antitrust people should be aware of potential issues as well as the FCC, and the FCC should be taking steps to make sure it has the data it needs to determine whether it needs to be doing anything to begin with," Bergmayer said. Levy prefers that government stay out of it. "Any time you get a regulator involved you're pretty much guaranteed for stuff to go downhill," he said, only half joking. "What they don't really understand is that sometimes this is OK. Basically, people need to argue. This is a relationship. No one can actually afford for this stuff not to work. Trust me." But some IP networks argue with each more than others, he acknowledged. At Hurricane Electric, "We have an open peering policy, which means if someone asks us to peer we have a very limited requirement on them in order to accept them as a peer." Hurricane Electric has so far been conspicuously absent from any of the recent public disputes. "We're boring," Levy said. "By design, of course, because we're not like the people who are in these disputes." Felten holds out hope that ISPs will become less likely to war with each other as they embrace over-the-top content—meaning the delivery of streaming video to any customer, even those who don't subscribe to the ISP's Internet service. ISPs that sell video outside their own networks will become just like Netflix and need other Internet companies to accept their traffic, the argument goes. They won't be able to throttle third-party traffic if they're worried about those same third parties throttling their own, Felten said. "If you do [over-the-top], you're not serving customers on your network only anymore, you're serving customers on everybody's network," Felten said. "If everybody else's network throttles your traffic, you have no business case. It can't work." Although Verizon is getting into over-the-top video through its Redbox venture, that's not the strategy most are taking (and it hasn't stopped Verizon from warring with Cogent over Netflix). Bergmayer is skeptical that Felten's theory will prove true in the US. "I've been hoping for a while that these guys would realize they could enter the over-the-top market," Bergmayer said. "But we've just seen things like TV Everywhere where it's essentially each ISP runs service just for its own customers and they sort of divide up the country like that." Complexity rules the Web Not every problem with streaming video can be traced back to peering agreements or caching disputes. One video expert told Ars that there are many components of video playback that can degrade quality, including the serving of advertisements. To load a pre-roll ad, "You have a third-party library involved which may or may not have been dynamically loaded. You may have third-party calls, there could be calls to several ad networks, and then the retrieval of that ad content and then the actual playback of the ad. There are a surprising number of steps to get just that simple preroll ad to play," said Frank Paolino, a senior service line manager at Akamai, a content delivery company whose businesses include technology for commercial video services. YouTube and Netflix pour lots of resources into upgrading streaming and video player technology to reduce performance lags. Last year, Google gave the YouTube player an overhaul that reportedly reduced buffering by an average of 20 percent. Each little performance gain is arguably more crucial in video streaming than in the loading of most types of webpages. "If you go to a retailer's website and their page takes half second to load, that is probably not perceptible to the human eye," Mike Palladino, Internap’s director of IP architecture and operations, told Ars. "If the graphics at the bottom take some time to load you probably don't notice that because you're looking at the stuff at the top that did load. Video and VoIP (Voice over Internet Protocol traffic) really exacerbate the symptoms we're seeing where demand outstrips supply." The Internet is a complex beast, with each packet of data trying to get to its destination through the most direct route, rerouting if necessary when something goes wrong. Visualization of the Internet, with lines drawn between nodes, each node representing an IP address. Wikimedia Commons When watching YouTube, "my computer opens a data stream connection to a YouTube server that is going to serve me this content," Palladino said. "That's one piece of it: establishing connectivity. The second piece is making sure there's a constant stream and I have good connectivity from start to finish. That's where we see things fall apart. If you're trying to watch an hour and a half video, the chances of you having zero connectivity issues from your host to that YouTube server for that length of time is almost zero. Network conditions fluctuate second to second, moment to moment. If someone takes a site down for maintenance in between, someone has a capacity issue on one of those interconnections," your video could stop or slow down. Take my cache, please It's because of problems like this that the likes of Google and Netflix offer to place their own equipment inside the data centers of Internet Service Providers. If Comcast, Verizon, and AT&T had YouTube and Netflix videos sitting inside their own facilities, video streams would take many fewer hops to reach customers. In addition to peering with network operators, Google has what it calls the "Google Global Cache (GGC)." This "represents the final tier of Google’s content delivery platform and is closest to users," Google says in its description of the service. "With GGC, network operators and Internet Service Providers deploy a small number of Google servers inside their network to serve popular Google content including YouTube. Google's traffic management system directs users to the node that will provide the best performance for the user. GGC can be located anywhere in an operator's network to maximize savings in backbone and transit bandwidth. Targeted deployment can reduce the number of route-miles traveled on an operator's network to serve Google traffic, further increasing cost savings for the operator." Google does not reveal which ISPs accept the equipment into their data centers. It's clear many do not, since ISPs have argued that Google and Netflix should be paying them, even though the caches are offered for free. Netflix's similar peering and caching service is called Open Connect. ISPs can peer with Netflix at up to eight Internet exchange points for free "or can save even more transit costs by putting our free storage appliances in or near their network," Netflix says. Frontier, British Telecom, TDC, Clearwire, GVT, Telus, Bell Canada, Virgin, Cablevision, Google Fiber, Telmex, and RCN are among those who have done so. "To us, it's really a black box that's run by Netflix," RCN VP of network services Peter Jacoby told Ars. "We do almost nothing except give it IP addresses. We put them in our data centers and that content gets a lot closer to our customers, so when they request a movie, typically the popular ones I'm told will stream from their closest cache and that eliminates a lot that can go wrong between them and Netflix." While this brings RCN additional costs in power, cooling, and colocation, it also reduces the amount of traffic RCN has to take from outside its own network. In addition, it gives subscribers more reliable streaming and access to "Super HD" and 3D content available only to customers of ISPs that use Netflix Open Connect. "To us, that's a win," Jacoby said. When Time Warner refused to partner with Netflix, it accused the video service of "closing off access to some of its content while seeking unprecedented preferential treatment from ISPs." Although Time Warner hasn't publicly demanded payment from Netflix, agreeing to settlement-free peering and caching would preclude Time Warner from demanding that Netflix pay to send traffic to Time Warner users. Felten warns that forcing companies like Netflix to pay all the last-mile ISPs to get their content to users would prevent future Netflixes from even existing. "If by default a peering relationship has to be paid by the online service provider, then the next Facebook will never exist, and the next YouTube will never exist because from day one they will be straddled with a cost they will be unable to bear," Felten told Ars. For today, at least one thing is clear. If you want an Internet Service Provider that has no qualms about peering with competitors and accepting caching services offered by third-party video providers, the biggest ISPs aren't your best bets. RCN, which competes against the likes of Verizon and Comcast in Chicago, Boston, New York, Washington, DC, and parts of Pennsylvania, knows that demanding money from Netflix isn't going to fly. Being the smaller player, RCN has to distinguish itself by offering better service, Jacoby said. Even though RCN provides video-on-demand content of its own, with rental fees for premium videos, that doesn't stop the company from making sure its customers get good YouTube and Netflix quality as well. "We'll be in line with our hand out if Comcast is getting paid [by Netflix] and Time Warner is getting paid," Jacoby said. "But there are network neutrality implications, there are a lot of questions I'd want to see answered before just going and shaking them down. What threat do we have? We're going to make service bad for our customers? We're not going to do that. Some of the other cable companies are willing to do that, but we're not going to. Being the little guy, that's the competitive position, and that's the differentiation."