Possible ways of implementing an HTTP proxy with priority mechanism

Question

I'm working on a TCP HTTP proxy: need to turn it in a priority proxy.

The standard proxy used to deal every connection with a fork and the child process handles the connection. Now I've implemented a priority queueing mechanism where packets coming from both sides of connections are queued and dequeued: packets coming from clients and remote servers are queued in my proxy, while packets are removed from queues and sent to clients or to remote servers. Packets are inserted with a priority algorithm of mine, and packets removed are guaranteed to be the ones with highest priority.

I thought three possible implementation of the receive packets from clients and servers and send packets with high priority to destination mechanism, but none of the three satisfies me or it has some conceptual issue.

1) The forking proxy spawns children processes, each of which must "talk" with priority queues, in the main process: this would require some kind of IPC, and for a fast exchange of data like a TCP connection IPC doesn't seem a good idea.

2) I may keep everything in the main: loop with a select over all the socket file descriptors (client and server sides) stored in a reading fd_set (so I can receive packets coming to the proxy and store them in priority queues). Packets that must be sent to client or server are first removed from queues, but the socket fd I will use to send the packet may block: so, make the socket non-blocking or spawn a thread that will send just one packet and then will die?

3) Third option is to use one thread per connection: again, each thread will have to talk to prio queues, inserting packets received from clients and servers in queues and sending packets taken from queues to its destination. But a CONNECT HTTP request is not a just simple receiving loop from server fd: it requires a select loop where packets received from the outside are queued, and in the meantime I have to send packets dequeued to destination.

I'm not the do my stuff for me guy, but I came up with only three solutions and can't figure other ways.

Working on Ubuntu 14.04, C++11.

Edit I'll provide more informations: the following method is called by the removal thread after removing the packet with highest priority from prio queues:

void dealPacket(Packet p) {
    int client = p.clientFD;
    int server;
    Request req = p.request;
    std::string payload = p.payload;

    // if there's no server file descriptor,
    // this is the request packet coming from clients
    if (p.server == -1) {
        // open remote server fd, connect to it 
        // and return the server fd
        server = openBindConnect(p.address);
        if (req == GET) {
            // send GET request to server
            if (sendGETRequest(server, payload)) {
                char data[1500];
                // while server sends packets, 
                // queue them up
                do {
                    Packet newPacket = craftPacket(data);
                    insertPacket(newPacket);
                } while (recv(server, data));
            }
        }
        else if (req == CONNECT) {
            // if connection was successful, must send 
            // a "200 OK" packet to client, then client and  
            // server will start sending and receiving
            if (send200OKtoClient(client)) {
                while (true) {
                    char data[1500];
                    FD_SET(client, &readset);
                    FD_SET(server, &readset);
                    // select() loop is used to check if 
                    // client or server sent packets; if so, 
                    // queue them up
                    select(maxFD, &readset);
                    if (FD_ISSET(client, &readset)) || if (FD_ISSET(server, &readset)) {
                        // receive Packets from client or server;
                        Packet newPacket = craftPacket(data);
                        // insert in queues
                        insertPacket(newPacket);
                    }
                }
            }
        }
    }
    // if this packets belong to an already existing connection, 
    // just send it (or close the socket pair <client,server> if 
    // this packet is the last one of one connection)
    else {
        server = p.serverFD;
        if (p.lastPacket) {
            close(client);
            close(server);
        }
        else {
            // send packet to destination
        }
    }
}

As you can see, it is a quite questionable method: there are send and recv, and this syscalls may block the thread, which is the removal thread, and in the meantime priority queues may have been filled while the removal thread is blocked because of send or recv.

It may be more convenient if the removal thread could just remove packets and let something else deal with them: for each dequeued packet, removal thread may spawn a thread that will open a new connection (if the packet is a HTTP request) or will just send the packet to its destination. But by doing so, lots of threads would born and die very quickly, thereby having more context switches than usual.

Answer 1

Packets are inserted with a priority algorithm of mine, and packets removed are guaranteed to be the ones with highest priority.

If you always get packets based on highest priority, there will be a load limit where low-priority packages will never be processed, will accumulate and lead to resource starvation in the proxy (though this depends on load and usage scenario - it may not be a problem needing to be solved in your case).

The solutions you propose all assume the processing loop should be common between client and server implementations, and handle packages transparently (regardless if they are client or server packages, and regardless of package priority - which is not visible/used in the code you used).

Consider the following changes (this is all speculative and subjective):

• use a task-based approach instead of a thread-based one.

• separate your execution priorities into multiple processing queues, by priority (high-priority packages, low-priority packages, cleanup tasks - like closing connections, whatever).

• these queues should accept functors as input, and return future results (see std::future, std::packaged_task, std::promise).

• Implement the queue execution/prioritization separately from the implementation of the operations.

Possible API (common to queues of all priorities):

/// perform operations passed through pointers on an internal thread
/// and when they are finished, set a promised result into a returned future
class queue
{
public:
    template<typename R, typename F> // F() should return a R instance
    auto add(F&& functor) -> std::future<R>;
};

Possible client code (inspired from your code above):

auto cleanupQueue = queue{ /* ??? priority arguments */ };
auto highPriorityQueue = queue{ /* high priority arguments */ };
auto lowPriorityQueue = queue{ /* low priority arguments */ };

/// snippet of code adapted from your question, for an example
void onClientConnect(Package p)
{
    // if connection was successful, must send 
    // a "200 OK" packet to client, then client and  
    // server will start sending and receiving

    auto response = highPriorityQueue.add([&]() {
        try
        {
            select(maxFD, &readset);
            ...
        }
        catch(const std::exception&)
        {
            cleanupQueue.add([]{ FD_Close( ??? ); }); // your socket here
            throw; // will get transmitted to "response" (this is the
                   // responsibility of the queue implementation you
                   // write; if you use std::packaged_task, you get
                   // this functionality already implemented
        }
    });

    // handle the promised response
    if(response.get()) // get blocks here until highPriorityQueue is finished
                       // will throw if the queued function threw
    {
        ...
    }
}

Another change to consider for your code:

• follow the SRP: the loop in your code does too many things (like handling of both server and client packages at the same time). If you separate it, it will suggest different design and will be easier to reason about.

A task-based approach (that is, using std::asynch and/or futures) should be superior to a thread-based approach here, because it gives you a more natural split for your tasks (and not necessarily prioritizing the entire algorithm execution, or prioritizing based on the data).