I need several duplex channels between two hosts. There are a number of advantages to establish only one TCP connection. But I doubt multiplexing would cause some inevitable problems. Will it harm performance or increase latency significantly? And what about memory usage and CPU usage? Is there any suggestion or caveat you'd like to give?
TLDR: The major drawback you might notice when multiplexing multiple channels on top of TCP (if you do it right) is an increased latency because of head-of-line blocking between the channels.
Corollary: If you don't care about latency you should be fine.
On the other hand using a single TCP connection “means less competition with other flows and longer-lived connections, which in turn lead to better utilization of available network capacity“.
Head-of-line blocking blocking over TCP
If you multiplex multiple channels on top of the same TCP stream, the channels might suffer of head-of-line blocking:
Head-of-line blocking (HOL) can occur when transport protocols offer ordered or partial-ordered service: If segments get lost, subsequent messages have to wait for the successful retransmission in the receiver queue and are thus delayed.
When you multiplex multiple streams on top of TCP you get HOL between the channels.
If channel A has filled up the TCP send buffer, you will have to wait before all of this data is received before any new data of channel B can effectively be transmitted to the remote application layer.
Examples of multiplexing over TCP
Channel multiplexing over SSH (over TCP)
A typical example of this is SSH. SSH can multiplex multiple channels (see
ControlPersist in OpenSSH). Using this reduces the cost of initializing a new SSH session (initial latency) but heavy transfer on one channel usually increases the latency/interactivity of the other ones (which does not happen if you use multiple TCP stream): if you are using a interactive sessions and start trigerring a heavy file transfer over the same channel, your session will start getting a lot less interactive.
Multiplexed HTTP/2 over TCP
HTTP/2 uses multiplexing of requests/responses over TCP in order to fix the HOL blocking. This feature is advertised in many articles and papers about HTTP/2. The HTTP/2 RFC claims:
HTTP/1.1 added request pipelining, but this only partially addressed request concurrency and still suffers from head-of-line blocking.
The resulting protocol is more friendly to the network because fewer TCP connections can be used in comparison to HTTP/1.x. This means less competition with other flows and longer-lived connections, which in turn lead to better utilization of available network capacity.
This is discussed in this LWN article about QUIC:
HTTP/2 was designed to address this problem using multiple "streams" built into a single connection. [...] it creates a new problem: the loss of a single packet will stall transmission of all of the streams at once, creating new latency issues. This variant on the head-of-line-blocking problem is built into TCP itself and cannot be fixed with more tweaks at the HTTP level.
Other multiplexing strategies
That's one of the distinguishing features of SCTP (multistreaming), you can have multiple independent streams in the same SCTP association and each stream does not block the other ones.
See SSH over SCTP — Optimizing a Multi-Channel Protocol by Adapting It to SCTP for the effect of using SCTP in order to avoid cross-channel HOL blocking in SSH:
SCTP only preserves the order of the messages within a single stream to mitigate an effect known as head-of-line blocking. If a message is lost, the subsequent messages have to be delayed until the lost one is retransmitted to preserve the order. Since only messages of the same stream have to be delayed, the number of affected messages after a loss is reduced.
By mapping the channels of SSH onto SCTP’s streams, the benefit of multi-streaming is made available to SSH, which is the mitigation of head-of-line blocking.
QUIC (multiplexing over UDP)
Another example, is the experimental QUIC protocol used for multiplexing HTTP over UDP (because multiplexing multiple streams on top of of TCP as HTTP/2 does suffer from HOL blocking):
QUIC is a new transport which reduces latency compared to that of TCP. On the surface, QUIC is very similar to TCP+TLS+HTTP/2 implemented on UDP.
Multiplexing without head of line blocking
Google’s QUIC protocol: moving the web from TCP to UDP presents a good overview of QUIC and HOL blocking when multiplexing channels on top of TCP.
A recent presentation claims that HTTP over QUIC improves latency but that the HOL-blocking improvement is a “smaller benefit“:
0-RTT, Over 50% of the latency improvement
Fewer timeout based retransmissions improve tail latency […]
Other, smaller benefits, e.g. head of line blocking
Note that while QUIC is described as “very similar to TCP+TLS+HTTP/2 implemented on UDP” it is in fact a general-purpose transport which can be used independently of HTTP/2 and might suit your needs.
Note: HTTP/QUIC si going to be standardized as HTTP/3.
I'd say you need to read the ZeroMQ Guide, the patterns it gives with the reasons and disadvantages are essential reading.
But otherwise, there's no problem with disconnecting the network channel from your application data delivery. You will have to take on the muxing and demuxing of data packets sent (and I would recommend a packet-based architecture here, not streaming data in a continuous flow) and the buffering of them on each end.
There should be little impact otherwise, you may need more memory - but only slightly - to buffer data, and a little more CPU as you are handling more code to lock and parse packets, but nothing significant. (unless you are writing something specialist that require a large throughput and performance is critical).
Yes, I've built a client-server database system using precisely this principle.
The channels multiplexed onto the one TCP connection each send packets of data, which are then split out to the respective recipients at the other end.
Hogging of the connection by one garrulous channel is done by the TCP connection sender doing round-robin selection of which packet to transmit from among the channels that have data ready to send.
To handle the case of one channel deciding to send a 1GB packet and locking everyone else out, the sender can choose to split a packet into chunks and only send one chunk before it gives another channel a turn. At the receiving end the reassembly of chunks into a packet is done before the recipient sees the packet.