My geeky self always gets excited when you meet someone who really understands the how and why of architectural choices in technology.
Recently someone explained to me that they had found the reason why some networks (backhaul for mobile, cable networks and others) did not perform as expected. The effective payload transferred can be reduced because the protocol overhead can be atrocious ( building TCP/IP on top of other protocols, which are themselves built on top of another etc.). The observed performance however was much worse than expected.
No, it was something else. A fatal combination of cheap memorychips and the wrong mindset.
The traditional telco mindset is to go for “5 nines”: absolute gold-clad reliability, no loss of data whatsoever. The designers of TCP/IP however designed their protocol for imperfection, expecting imperfection (packet loss, link loss, erratic throughput) and using the imperfection in the algorithms for congestion control. In other words, if you want to run at maximum (dynamic) performance you have to exceed the limits regularly in order to know where they are.
Deep down in the network gear one uses memory chips as a buffer for incoming data between the network and the application. If the buffer is full, you lose a sent packet. TCP/IP “sees” the loss because the reception of the packet is not acknowledged (ACK) by the receiver to the sender.
But what happens if the traditional telco mindset and the cheap big memory chips of today meet each other? The buffer is set to be very big “so you don’t lose packets”. Who cares, memory is cheap.
The effect is that sometimes a lot of data can be held waiting in the buffer for the application. No ACK’s (acknowledgements) are returned to the sender (yet). TCP goes crazy and thinks a lot of data is lost, reduces the sending rate and starts to retransmit all the data again, bringing the throughput to its knees.
The fix was easy: reduce the buffer to a small size so TCP/IP gets its desired feedback quickly. Performance was excellent again as a result.
The lesson is a deep one about the conceptual difference between striving for perfection on one hand and (elegantly) designing for imperfection on the other hand.
The difference between control and governance in a nutshell.
