Hyperconnectivity: November 2009 Archives

Congestion Neutrality

Posted by Herman | November 30 | 2009 11:34 AM | permalink

(This is a co-post of Herman and Benoit at Fiberevolution)

With the debate on net neutrality in full swing in the US, we've been hearing about Bandwidth Hogs again. 'Bandwidth Hog' is a sound bite that conveys a strong emotion: you can virtually see the fat pig chomping on the bandwidth, pushing back all the other animals in the barnyard with his fat pig shoulders all the while scrutinizing with his shiny piggy eyes to see if the farmer isn't around...

As analogies go this a quite mean one, obfuscating the technological reality and design choices telco's have made. (For anybody who is not into the bits and bytes you can find an overview over here @ Fiberevolution, including an offer you can't refuse).

What is the reality?

Lets take a DSL network for example.

When you purchase an Internet access line advertised at 8 Mbps download you get a dedicated connection rated at that speed to a concentration point (switch house, telephone exchange). You cannot "hog" that connection because it is all yours, you are the only user of that connection. The telco combines all dedicated connections in the concentration point and connects them in a shared high speed link to the nearest Internet Exchange (or peering point).

To save money they overbook that link, up to 40, 50 or a 100 times. In practical terms: if everybody tries to use their Internet access fully at the same time, they will experience 1/40th (or less) of their maximum advertised bandwidth because of the congestion in the shared link. The shared link is therefore a bottleneck, a chokepoint where congestion will rear its head.

Fortunately the chances that we all will try to use our access line fully at the same time are quite low. Experience has taught us in the past with the usage patterns of these days that as telco you could get away with these overbooking ratio's.

When users started to complain the telco added some capacity in the chokepoint. You can find in the fine print that you get a "best effort" connection, which is the legal translation of the reality described above.

It did and does make sense to overbook the shared link because it reduces the cost of an Internet connection. But the big elephant in the room is the fact that all telco's compete on advertised maximum speed and hardly anybody dares to be very clear about this bottleneck.

Time however is running out because usage patterns have changed over time (more video), expectations have been raised, and the bottleneck is getting noticeable. Some telco's increase their capacity so to adapt to their users, others seem to see this as a good excuse to prevent the appearance of Net Neutrality regulations and introduce traffic rationing. After all, with artificial scarcity you can make more money....

Going back to the "hogging" theme, does a heavy user hog the chokepoint?

According to the experts we consulted the following is a good approximation of the complex reality, first of all for a DSL network.

Lets assume we have a 1 Gbps shared middle mile link for a number of users, all having bought a 10 Mbps rated Internet access line. In the beginning only 1 user is active. He initiates a download of a big file of a server with no speed limitations (makes it easier to explain). The server finds no limitation in the link, sends data at 1 Gbps and quickly saturates the receiving buffer of the particular access line modem (DSLAM for example) of the user. The effect is that packets are dropped . Which is a signal to the sending server to reduce its rate to the level where only now and then a packet is dropped (testing the maximum speed possible of the access line).

This is the basic congestion mechanism of TCP.

Other users start doing the same thing, the same happens until the shared link starts to get fully utilized and more. The congestion in the shared link leads to the shared link receiving buffer being flooded, dropping packets as well.

The result is that all servers sending data reduce their speed simultaneously until the number of dropped packets is acceptable. The dropping of packets is a statistical phenomenon, with every user having the same chance.

The effect is a gradual reduction of speed. [ Update: but also to an increase in latency which hurts perceived performance, see http://www.dadamotive.com/2009/12/congestion-neutrality-2.html]

The straightforward implementation of TCP will lead to a more or less equal distribution of bit-rates per TCP/IP session. How many sessions do your create as a user? Very few with email, maybe (but not necessarily) a lot by using Bit-Torrent or by surfing the web.

That having said, a more complex implementation of TCP might do it differently and reduce the bitrate for all session coming from one IP-adress.

It is all a design choice.

For a cable operator the bottleneck is already inside the hybrid fiber-coax access network itself. The access channel is itself shared with others and the cable modems reduce your maximum speed if its gets busy. The effect is that the users who have bought the high access speeds feel the reduction first because the upper limit is reduced to the same level for everybody (If you have bought a low speed access subscription you will notice it the last). So the congestion in the middle mile leads to the highest paying users complaining first. Harsh maybe, but very straightforward. If you assume that the "bandwidth hogs" have the fastest connections, they will be the first to get the brakes on if congestion occurs.

So hogging does not exist as claimed, only chokepoints and designs that companies are hiding from sight.

The thing that bothers us is the lack of transparency on these crucial design issues. ( Benoit is offering free consultancy on datasets if telco's will generate them, so let's see if someone will take it up.).

Instead of Network Neutrality rules we might want to start with Congestion Neutrality rules.

No fiber please

Posted by Herman | November 26 | 2009 6:37 PM | permalink

A seemingly unrelated and innocuous taxation may explain a part of the (lack of) broadband development in the UK, especially outside the bigger cities.

Lit fibers are subject to a property tax (unlike copper wires used for telephony or DSL).

The VOA (Valuation Office) sets the rateable value.

You can expect to pay for backhaul or corporate networks

£0.28 per metre per fibre per annum for 2 fibers,

£0.17 per metre per fibre per annum for 4 fibers,

The rate scheme drops quickly with the number of fibers in a route to £0.05 per metre per fibre per annum for 40 fibers.

£7.50 per home passed per annum for fibre access networks (to be finalized).

" As NGA will be mainly the replacement of existing copper infrastructure, the VO considers that the level of value for residential NGA connections will be similar to the £7.50 per home passed adopted for cable TV access networks, the nearest comparable network currently offering broadband services".

The rating scheme penalizes small operators and companies that start their business: their average cost is much, much higher than large existing corporations with a lot of fibers and long lengths. The ones with a lot of fiber can add fiber with low marginal costs of taxation, new entrants have a hard time.

What about smaller villages?

Lets assume a minimum of 2 fibers in a redundant ring to a village with 1000 homes where 50 % of the people are dying for broadband access, approx 30 km away from the nearest backhaul aggregation point.

The tax rate is £0.28 per metre per fibre per annum for backhaul of 2 fiber, 60.000 meters (redundant ring) and 2 fibers lead to a tax bill of £ 33.600 per year, or £ 67.2 per connected user (50 % penetration). Auch.

No wonder you are on your own if you live somewhere a bit off the centre of a city. The business case get harder adn harder just because of taxation designed in 2000. The backhaul networks are probably minimized in number of fibers and capacity, shared as much as possible to reduce the costs, leading to high overbookingsratios and lacklustre response times of websites. Could that again be a reason why broadband adoption outside the cities is relatively low?

It is hard to get a causal relationship proven, but one starts to wonder....

Openreach: to boldly go where you don't want to be

Posted by Herman | November 25 | 2009 6:55 PM | permalink

The presentation of Openreach during the "Fibre to Britain" conference on their high speed broadband roll-out to 10 million homes in 2012 led to mixed feelings.

On one hand you have to admire the engineering and management needed to ramp up the rollout to a level of 800 VDSL cabinets per month, each serving 300+ customers. That's 250.000 homes per month. The plans include now FTTP (GPON), for approx. 25 % of the footprint, and Korean-style VDSL-inside-the-building.

The engineering challenges are not to be underestimated. During the test phase 6 % of the cabinets were damaged for various reasons, including people backing up their cars into the cabinets. The GPON test showed that the standard fiber cables are too strong for areal deployments in the last drop: regulation demands that a high vehicle running into the cables does not take the poles with them, the cables must break first. So miniducts are used for the last drop.

But the resulting broadband offerings (especially VDSL) did not generate any enthusiasm amongst the audience. The communication providers define the final retail offerings, but capacities of 20/2 Mbps, or "up to" 40/20 Mbps where IP-TV takes a large chunk of the available bandwidth make it hard to create compelling arguments for users to switch. No prices were disclosed, but the expectations of the penetration rate by Openreach were quite low: will there be a market ? it still had to be proven said the presenter. Low penetration expectations mean high unit prices if you do not want to lose your shirt.

One exasperated listener defined it as "fear will get you a self fulfilling promise: a failure".

You could almost feel the tension which must exist in Openreach: between the engineers trying to do a great job, the financial people who ask for proof of a market before committing scarce financial resources, and managers trying to compromise between the pressure to move with the trend and their new role as infrastructure providers.

Lambda to the Home and path dependencies

Posted by Herman | November 21 | 2009 4:13 PM | permalink

As the debate on Fiber-to-the-Home shifts to how to manage the transition from copper/coax to fiber, there is more attention for the future upgrade paths. First of all the attention will shift to the backhaul/middle mile networks which aggregate the traffic generated by a large number of users. As access speeds grow the capacity of these networks will have to grow. A well established upgrade path.

Even in the access networks there is already lot of development to increase the access speeds even more, up to the Gbps territory and beyond. The active Ethernet technology is quite stable but in GPON there is a proliferation of new standards, not all of them compatible.

Digging deeper into these options it turns out that not all fiber access networks are the same. Topology matters and creates path dependencies.

As an example: if you have deployed a passive optical topology with deep splitters (close to the customers) with a high split ratio (> 32), you cannot upgrade from GPON to 10GPON (4 times faster) without investing in the fiber plant or losing speed. You will have to reduce the split ratio which means laying new fibers to the splitter.

The alternative is to switch to a different technology, wave division multiplex (WDM), sending multiple colors over one fiber . Applying WDM requires ripping out the splitters in the field and replacing them with a wavelenght filter that separates the colors to individual customers. The filter selects one wavelength per user, making the fiber plant dedicated to WDM.

Path dependencies all over the place, unless you have a point-to-point topology.

Last night I had the pleasure to have a conversation with Johan Heneas, CTO of INS Communications (Norway). INS is a systemintegrator specializing in WDM technologies. His explanation of modern WDM technology led to an unexpected revelation: it will allow cost-effective "Lambda to the Home" in point-to-point topologies in dense urban area's.

The newest thing in WDM is a passive filter for many wavelengths (somewhere between 16 and 64). If you increase the number of wavelengths the filter has to be sharper and sharper. These passive filters are built from glass and are sensitive to temperature changes. Which becomes a problem if you increase the number of wavelengths and place the filter outside in the field. The solution so far was to add a heater and stabilize the temperature, requiring electrical power and something that can break in the field. Yuk....

The clever solution is to add another material to the filter which temperature sensitivity is exactly the opposite of the glass, compensating the changes. And it works. According to Johan their operational experience with these filters in the outside fiber plant (in the Norwegian weather) are excellent.

The second trick is to have automatic tuned lasers in each customers modem for the upstream signals. Each customer has its own separate wavelength for upstream bandwidth. It is possible to use the same wavelength as the downstream link but you run in to problems with reflections and the like when the speeds are increased.

To prevent a logistic nightmare (you don't want to mix up the right combination of upstream and downstream wavelength per user) automatic tuning has been developed. A broad spectrum lightsource is filtered at the head-end in (for example) 40 specific wavelengths. These wavelengths are adjacent to the sending lasers wavelengths so they automatically pass through the same filter to the right customer. That seed-light is a tuning trigger for the upstream laser which "clicks" to the right wavelength. Clever.

The technology is not yet standardized and still more expensive per user than GPON or active Ethernet (depending on the implementation factor 2-4). With larger volumes the gap will become smaller I guess.

This higher cost can be justified in rural areas where the distances to a POP or aggregation point are long (30 km or more). A standard shared topology (PON) runs into problems (not enough light) at larger distances, requiring a lower split ratio, more fiber and a higher average costs of OLT's ("head-end equipment") per customer. WDM has a much larger optical power budget (15 db more), allowing for much longer distances.

The surprise came when we applied the technology to a situation like in Amsterdam, in a dense urban area with a point-to-point topology. What could this WDM possible bring in this environment? A bypass of the middle mile , removing the limitations.....

The filter is situated in a POP (with 10.000 fiber connections terminating in a POP). You patch a customer specific fiber to the filter and patch a backhaul fiber to the other side of the filter. The backhaul fiber can be 10-20-30 km of length before you terminate it in a WDM line card. So up to 40 individual customers get their own separate lightpaths , bypassing the normal backhaul concentration links.

The WDM line card could be situated in an institution (science or research or government or media or high security environment) creating a dedicated ultra-high-speed optical Lambda-to-the-home.

If you live in Amsterdam the WDM line card could even be located near the AMS-IX, giving a dedicated link from the home to the biggest Exchange in the world. Mindboggling.

I can very well imagine that there is a market for this kind of service offerings.

And again an example what the advantages are of minimizing path dependencies in your topology......

The definition of broadband

Posted by Herman | November 20 | 2009 8:21 PM | permalink

There is no single definition of broadband available. Most definitions express it in bandwidth: 200 kbps, 1 Mbps, 5 Mbps, 100 Mbps? The diversity is very large, depending on who you ask it.

In my opinion the main problem is that we try to combine two disjunct objectives in one number.

The first objective is to define a social minimum. Broadband access is becoming a utility like water and electricity, it is hard to function in this society without it. So it makes sense to define what the acceptable minimum acces specifications are, and to define policies how to make sure everybody can have acces to it.

One could argue that a 512/512 kbps access capability would be a social minimum nowadays, to be upgraded over time.

The second objective is aspirational. At what level of (affordable for median to higher incomes) broadband access are we competitive internationally, can we participate in the forefront of services and possibilities which strengthens our economy?

This level will probably now be at 100/100 Mbps, rising to 1000/1000 Mbps in the near future.

The twain shall never meet. So lets solve this problem by defining them separately.

Overestimated

Posted by Herman | November 19 | 2009 9:27 PM | permalink

In June of this year the so-called "Hamburg Declaration" was released. A cry from 166 international publishers to get legal protection, to get paid for others linking to their sites.

Numerous providers are using the work of authors, publishers and broadcasters without paying for it. Over the long term, this threatens the production of high-quality content and the existence of independent journalism.

148 Of the 166 publishers are german, so The Reach Group GmbH (TRG) did a test.

Google is often cited as one of the main villains linking "for free" to this costly cintent.

So what happens if you would imagine that all these sites would block Google from accessing their content, how empty or different would the top-ten of search results become?

It turns out that only 5 % of the top-ten search results of Google links to the domains of the publishers, and the links are not very relevant.

The domains of the publishers contribute to 4 % of the Google index for Germany, much less than for instance Wikipedia. Apparently the content is hard to find for search engines. Even in Google News they are hard to find.

Apparently these publishers overestimate their own importance on the web.

Next Generation Connectivity (3)

Posted by Herman | November 15 | 2009 9:53 PM | permalink

The Berkman report has created a lot of waves in the USA. Various lobbyists have attacked the report to undermine its credibility.

The main author, Yochai Benkler has written a great response to all his critics which you can read over here (pdf). I love the last part:

In 2002 France still had a very weak broadband market. One would not have predicted that France Telecom's offer, the most expensive among major offers in the French market, would, within six years, be for speeds twice as fast as those offered by Verizon for a little less than one-third the price, or would be offering speeds several times as fast as those AT&T offers at roughly the same price. Similarly, KDDI's investments in power company lines in Japan, complemented with unbundled DSL, should surprise those who predicted that open access policies would cause disinvestment from competing platforms. Similarly, the fact that NTT is offering price/performance ratios that put France Telecom's to shame, but are identical to the offer by Japan's innovative entrant, Softbank, also strongly suggest the presence of a highly competitive market.

Do I have anything to comment on in this report? Yes, some additions are listed below.

I understood from my Japanese contacts that one of the stimulating factors was the unbundling price of copper lines. Just above the OPEX of maintaining the lines. So the owners of infrastructure actively sought new revenue streams by investing in better infrastructure.

Quite the opposite of the Western world where the enormous cashflow out of fully depreciated copper lines and cable created financial addiction.

There is a growing understanding in Europe that the financial markets will prevent incumbents (including cable) from investing in NGN until all other options are exhausted (see the 70 % hurdle). The comfortable unbundling line fees do not help. The reason is: a gigantic cashflow and for some incumbents still quite high debts remaining out of the crazy 3G and "world domination" era.

EBITDA margins can be as high as 55 % of revenue, quite addictive.

Without a very pressing need (life threatening), why would you allow a company to pour that cashflow into an infrastructure? Without rocksolid proof that there will additional ARPU (and worse, more indications like Free that prices will drop?) (http://www.fiberevolution.com/2009/10/of-dinosaurs-evolution-and-suicides.html is a good read on the latter)

So succesfull rogue new entrants (like Free or Reggefiber) and municipalities create the pressure that (willing) incumbents can use to convince the financial markets that they need to move also or face trouble. The roll-out of DOCSIS 3.0 which is an answer of cable to these new entrants increases the pressure on telco's.

Another part of the pressure on telco's is cable voice. The idea of facility based competition originates partly from the days telco's had a monopoly on the fixed voice infrastructure (high utilization, all costs covered) and cable had a monopoly on TV-distribution (high utilization, all costs covered). Internet broadband was the area of competition, at marginal costs. An artifact created by the safe harbours of monopolies.

Cable was able to attack the "safe harbour" of telco's by adding VOIP, telco's tried to respond by adding TV (unsuccessfully) so they are more exposed.

Why is sharing the access infrastructure good business?

The financial lever of higher utilization of the infrastructure's "sunk costs" is quite significant.

The secondary effects of a vibrant ecosystem where consumers have a choice is that is becomes more attractive for everybody (consumers, developers, added services). The tertiary effect not often mentioned is the reduced costs of innovation for the market leader. It is much cheaper to let 1000 flowers bloom (financed by others) and follow in the mainstream what is succesfull.

Fiber to the home has an advantage over FttC. It is quite easy and cheap to add broadcast TV via a 3rd wavelength (as is done by FiOS, in Japan en many other places). It does NOT compete for bandwidth with Internet access, does not influence latency, is cheap and reliable.

Reliability expectations for TV are way higher over here than expectations for Internet access. There are hilarious stories known of the local police picking up engineers at home saturdaynight to repair TV distribution because a mad crowd was badgering the police (popular football game on live TV).

The same spectrum as cable can be used (50 - 900 MHz), same modulation techniques, same gear. Many TV-sets sold nowadays in the EU have an embedded DVB-C tuner which decodes digital (HD-)TV without a separate settopbox, quite convenient. You even can use without extra costs the spectrum up to 3 GHz for more channels or higher bandwidth HD.

Broadcast is cheaper and (currently) more reliable than IP-TV.

Broadcast TV adds ARPU to a telco's revenue and is a new competitive weapon, justifying investments in NGN.

Cable is responding however by creating exclusive distribution deals with HD-channels. For policy makers this is a can of worms because you enter the copyright legal arena (exclusivity is based on copyright law).