I'm working on a TCP HTTP proxy: need to turn it in a priority proxy.

The standard proxy used to deal every connection with a fork and the child process handles the connection. Now I've implemented a priority queueing mechanism where packets coming from both sides of connections are queued and dequeued: packets coming from clients and remote servers are queued in my proxy, while packets are removed from queues and sent to clients or to remote servers. Packets are inserted with a priority algorithm of mine, and packets removed are guaranteed to be the ones with highest priority.

I thought three possible implementation of the receive packets from clients and servers and send packets with high priority to destination mechanism, but none of the three satisfies me or it has some conceptual issue.

1) The forking proxy spawns children processes, each of which must "talk" with priority queues, in the main process: this would require some kind of IPC, and for a fast exchange of data like a TCP connection IPC doesn't seem a good idea.

2) I may keep everything in the main: loop with a select over all the socket file descriptors (client and server sides) stored in a reading fd_set (so I can receive packets coming to the proxy and store them in priority queues). Packets that must be sent to client or server are first removed from queues, but the socket fd I will use to send the packet may block: so, make the socket non-blocking or spawn a thread that will send just one packet and then will die?

3) Third option is to use one thread per connection: again, each thread will have to talk to prio queues, inserting packets received from clients and servers in queues and sending packets taken from queues to its destination. But a CONNECT HTTP request is not a just simple receiving loop from server fd: it requires a select loop where packets received from the outside are queued, and in the meantime I have to send packets dequeued to destination.

I'm not the do my stuff for me guy, but I came up with only three solutions and can't figure other ways.

Working on Ubuntu 14.04, C++11.

Edit I'll provide more informations: the following method is called by the removal thread after removing the packet with highest priority from prio queues:

void dealPacket(Packet p) {
    int client = p.clientFD;
    int server;
    Request req = p.request;
    std::string payload = p.payload;

    // if there's no server file descriptor,
    // this is the request packet coming from clients
    if (p.server == -1) {
        // open remote server fd, connect to it 
        // and return the server fd
        server = openBindConnect(p.address);
        if (req == GET) {
            // send GET request to server
            if (sendGETRequest(server, payload)) {
                char data[1500];
                // while server sends packets, 
                // queue them up
                do {
                    Packet newPacket = craftPacket(data);
                } while (recv(server, data));
        else if (req == CONNECT) {
            // if connection was successful, must send 
            // a "200 OK" packet to client, then client and  
            // server will start sending and receiving
            if (send200OKtoClient(client)) {
                while (true) {
                    char data[1500];
                    FD_SET(client, &readset);
                    FD_SET(server, &readset);
                    // select() loop is used to check if 
                    // client or server sent packets; if so, 
                    // queue them up
                    select(maxFD, &readset);
                    if (FD_ISSET(client, &readset)) || if (FD_ISSET(server, &readset)) {
                        // receive Packets from client or server;
                        Packet newPacket = craftPacket(data);
                        // insert in queues
    // if this packets belong to an already existing connection, 
    // just send it (or close the socket pair <client,server> if 
    // this packet is the last one of one connection)
    else {
        server = p.serverFD;
        if (p.lastPacket) {
        else {
            // send packet to destination

As you can see, it is a quite questionable method: there are send and recv, and this syscalls may block the thread, which is the removal thread, and in the meantime priority queues may have been filled while the removal thread is blocked because of send or recv.

It may be more convenient if the removal thread could just remove packets and let something else deal with them: for each dequeued packet, removal thread may spawn a thread that will open a new connection (if the packet is a HTTP request) or will just send the packet to its destination. But by doing so, lots of threads would born and die very quickly, thereby having more context switches than usual.

  • Another popular design is to have as many workers (threads that consume from a (priority) work queue) as you have CPUs. I/O by workers needs to be non-blocking and all that entails regarding I/O completion work, so you don't loose a CPU on I/O. It is more complex than spawning new threads for blocking I/O completion activities, but avoids that overhead. – Erik Eidt Nov 7 '16 at 17:06
  • You are talking of std::thread::hardware_concurrency()? The problem is my queueing algorithm has already two working threads (one is looping for perpetual removal, the other is looping for avoiding starvation), both started at the beginning of the main. So, if std::thread::hardware_concurrency() returns 4, my thread pool would be of 2 threads, since 2 are already taken by the queueing algorithm. – elmazzun Nov 7 '16 at 17:21
  • Not to speak for @ErikEidt but the one worker per thread is sort of a rule of thumb. If the workers are idle for significant portion of the time (e.g. during IO wait) you can run other things on same threads. Theoretically you can run 3 threads that are idle 1/3 of the time on 2 CPUs: 3 * (2/3) = 2 In reality it's never that clean so tuning is important. – JimmyJames Nov 7 '16 at 18:09
  • One tweak to what @ErikEidt is proposing (a good idea) is to purposely add more workers than you have CPUs and make some workers threads have a lower priority. These lower priority workers will then handle the lower priority work. One thing that might make this easier is to have separate queues based on priority. One issue with a single priority queue is that if you have no high-priority work at a given moment, all your workers could start working on low-priority work right as high-priority work comes in. – JimmyJames Nov 7 '16 at 18:16
  • Please see the updated question. – elmazzun Nov 8 '16 at 15:22

Packets are inserted with a priority algorithm of mine, and packets removed are guaranteed to be the ones with highest priority.

If you always get packets based on highest priority, there will be a load limit where low-priority packages will never be processed, will accumulate and lead to resource starvation in the proxy (though this depends on load and usage scenario - it may not be a problem needing to be solved in your case).

The solutions you propose all assume the processing loop should be common between client and server implementations, and handle packages transparently (regardless if they are client or server packages, and regardless of package priority - which is not visible/used in the code you used).

Consider the following changes (this is all speculative and subjective):

  • use a task-based approach instead of a thread-based one.

  • separate your execution priorities into multiple processing queues, by priority (high-priority packages, low-priority packages, cleanup tasks - like closing connections, whatever).

  • these queues should accept functors as input, and return future results (see std::future, std::packaged_task, std::promise).

  • Implement the queue execution/prioritization separately from the implementation of the operations.

Possible API (common to queues of all priorities):

/// perform operations passed through pointers on an internal thread
/// and when they are finished, set a promised result into a returned future
class queue
    template<typename R, typename F> // F() should return a R instance
    auto add(F&& functor) -> std::future<R>;

Possible client code (inspired from your code above):

auto cleanupQueue = queue{ /* ??? priority arguments */ };
auto highPriorityQueue = queue{ /* high priority arguments */ };
auto lowPriorityQueue = queue{ /* low priority arguments */ };

/// snippet of code adapted from your question, for an example
void onClientConnect(Package p)
    // if connection was successful, must send 
    // a "200 OK" packet to client, then client and  
    // server will start sending and receiving

    auto response = highPriorityQueue.add([&]() {
            select(maxFD, &readset);
        catch(const std::exception&)
            cleanupQueue.add([]{ FD_Close( ??? ); }); // your socket here
            throw; // will get transmitted to "response" (this is the
                   // responsibility of the queue implementation you
                   // write; if you use std::packaged_task, you get
                   // this functionality already implemented

    // handle the promised response
    if(response.get()) // get blocks here until highPriorityQueue is finished
                       // will throw if the queued function threw

Another change to consider for your code:

  • follow the SRP: the loop in your code does too many things (like handling of both server and client packages at the same time). If you separate it, it will suggest different design and will be easier to reason about.

A task-based approach (that is, using std::asynch and/or futures) should be superior to a thread-based approach here, because it gives you a more natural split for your tasks (and not necessarily prioritizing the entire algorithm execution, or prioritizing based on the data).

| improve this answer | |
  • The "promised response" over the last if is the packet removed from a priority queue? Because if it is from a GET request, all packets from server must be enqueued (missing in your code snippet); if it is a CONNECT request then, like you did, I have to enqueue them anyway and this, I do. My problem is sending the packets retrieved from queue: it may be my ignorance speaking, but I don't see the solution. GET or CONNECT, I read packets coming from both client and server and I enqueue them, that is easy: the matter is sending them. – elmazzun Nov 10 '16 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.