Before jumping to my questions, let me explain my project's background.
I am part of a team that organize the network of a famous LAN event. Most of the games nowadays actually don't run locally but use some remote servers instead. We do have one gigabyte connection with a single public IP at our disposal and can't afford any other technical solutions. Sadly, most of the game servers/network protections will react really really badly when suddenly 500+ users log in a short from a single public IP. A hosting company gracefully lends us a so-called cloud composed of some devices running ARM CPUs (4 cores) and we hence have a huge bunch of public IPs. Until now our trick consisted in using OpenVPN to tunnel some connections by the cloud, but OpenVNP isn't optimal for some reasons especially on our ARM CPUs: OpenVPN runs on one saturated core (mono-threaded) with a depressing low bandwidth.
Firstly we don't care about the security or the high resilience of OpenVPN, we want a bare tunnel. Our first actions seeing these bad results was to tweak, using the config files, all the CPU consuming options (encryption...), use UDP, start one instance per core... Even with all these actions, the throughput is a bit disappointing. This is where we start to think about creating a highly specialized tunnel in C++ for our needs. No big engineering like OpenVPN that does its job perfectly fine as a general purpose VPN.
First side-question: is there already any good solution that solves our problem? We don't want to recreate the wheel.
We started working on a userland version of the tunnel using linux tun/tap devices. Does a kernel module looks like a brighter idea? Technologies speaking, I will most likely use C++11 and boost.
I have come up with two architectures, both are divided in roughly 4 components: - One that receives/sends some data from the network interface. A UDP socket. - One that allocate a some memory chunks for the packets that are received or sent. - One that route the packets. - One that write on the tun/tap interface.
My first architecture would use boost::asio for the UDP socket, the router and the interface (note: one interface can be described in multiple file descriptors that can be encapsulated for asio). Each of these components would be have their own asio::strand for avoiding racing conditions and all the nasty threading related problems. I would spawn 4-8 threads running the event loops. Asio and the asynchronous methods looks gorgeous and ease the message passing between my components. But first epoll/select might not be super efficient for a single UDP socket and few stream descriptors. We are not dealing with the C10k problem here, no need to scale for the clients number as a tunnel is only 1 <==> 1. Second, I fear that asio message queues are not really efficients, we want a high bandwidth and bandwidth is tightly linked with latency. Last but not least, asio has some inner locking mechanisms that I fear a bit.
My second architecture would use 3-4 dedicated threads (one per components) spinning like crazy and doing the respective jobs. Blocking IO or shouldn't matter. The messages passing between the components would be using some lock-free queues. This solution looks harder to code. The throughput should be really high and stable (resistant to spikes/bursts) but it also means that all the server resources will be burned with our threads spinning like crazy. Since our tunnel won't do much more than routing, the time might be spend more on the IO than the CPU actually.
As for the allocation components: I thought to use a pool of objects (packets) combined with some shared_ptr like this: https://stackoverflow.com/questions/2911154/custom-pool-allocator-with-boost-shared-ptr
Or to use a ring buffer, some packets could be dropped in the process thought.
Do you have any other idea for an efficient architecture, or any review or preference for the previously mentioned ideas? We are trying to find the best between an elegant and an efficient solution.
Any other hint for an efficient tunnel is welcomed! (Like not using TCP over TCP).