How to combine epoll + multithreading in an HTTP server

Question

I'm building an HTTP server in C using epoll and pthread. The idea is to have an event loop in the main thread monitoring file descriptor and a thread pool to handle computationally-expensive operations.

What I'm struggling is that what type of operation should be done by an event loop and by a thread pool. Here is what I expect to be done:

• My epoll event structure will store a request struct and a response struct.
• After adding new file descriptors to monitor (EPOLLIN), the server will read from a file descriptor if there is a new request, and store to the request field in the epoll event
• After reading the request, the event loop will push it to a queue. A thread pool will take requests from the queue and process all the requests (assuming all static files).
• After processing, the thread will store a response back and switch the event flag to EPOLLOUT. The event loop will monitor and send out the response.

Here is what I've seen in many examples:

• Accept a new connection and add the file descriptor to the epoll interface.
• If there is an event, except from accepting new connections, the thread pool will handle data from the file descriptor.

In a nutshell, I think that threads should only handle operations such as reading large static files, and the event loop should handle network and socket operations such reading from or writing to sockets. What do you think?

---

(I updated this post with my own solution)

Answer 1

epoll only notifies you that an specific fd is able to receive data, or has data to read. then reading from non-blocking socket or writing into it doesnt block your execution or waits til data is transfered.

reading in blocking IO result in: if data is available your buffer is filled with data, otherwise thread goes to sleep til data is available or a failure detected

writing in blocking IO result in: if able to write, then you write and thread goes to sleep til data is written, otherwise thread goes to sleep til able to write

but in non-blocking IO, thread never goes to sleep, you only receive notification of availibility for reading and writing for specific fd. and for detecting "its enough to read or write", read or write will return -1 and errno is set to EAGAIN or EWOULDBLOCK.

dont worry about errno, its per thread, your execution wont mess.

reading or writing is the same for kenel in any situation, no matter its blocking or non-blocking. its all about reducing number of context switiching by reducing number of threads. hence you receive more performance.

when you are in a situation you need to do any async task, you need more playing with states than doing traditional jobs.

for example:

in a blocking IO for parsing http requests you read as much as a full http header is parsed then step into next phase. parsing in blocking IO is easy, because you dont miss which part of your parsing phase you were, as you dont leave that part.

but the game in non-blocking IO is different, because everytime you receive a partial data, consider receiving data as below:

first

GET /

then

index.php http/1.1

then

\r\n\r\n

so how do you parse that data?

you have two choices:

1. store as much in a cache til you receive double \r\n then parse with your previous algorithm

2. start parsing from a previous stored state, if done return, if not, store state for next data arrival ...

receiving data in partial as above is a rare case, but in busy networks with some custom configs you will face such a case.

but the problem is not parsing headers, its all about you have to do everything in partial as your data is ready partially. so you need to store states of the place you previously were in, for any connection.

so when you are doing non-blocking IO, no matter you are doing all things in one thread or pass to other threads to do stuffs, anyway you are lost in lots of partially done stuffs.

so you need to properly store states of anything, yes anything. and continue from that state. and that makes your code more complicated as the problem is complicated.

we wait for another nginx with more performance from you ;)

Answer 2

It seems you are trying now to find a simple answer to a complex multi-axis question under conditions that nobody knows except you. We donʼt know how exactly you organize network interaction - for example, how many copyings are used there (they are CPU-bound part of a network server, especially for a text one like HTTP). We donʼt know how your inter-thread interaction is organized. There would be loads of other factors on which a solution really depends.

Well, there are some principal considerations which should affect the design.

As you mentioned, disk reading is, with default functions or mmap, blocking. If you use it that way, a thread pool is a solution. But, if you mentioned epoll, this definitely means Linux. Having Linux you may utilize io_uring. This is a rather new tecnhnique, but seems already attaining the stable state. But even it the cost to parse requests, generate response headers, call handlers for URL parsers, find file in FS, etc. - will be mainly unsplittable into small asynchronous tasks.

You said - to thread pool for large static files? Well, it seems you mean that cost of a task offloading to a pool could be greater than reading a single file. But you may fall into IO overload. Blocking of the main thread with a delay of IO operation that was postponed for even a few tens of milliseconds due to scheduler's insane decision... seems this is not what Iʼd expect from such a server. Better is to decide that any new connection goes to a thread pool, but - it is crucial - it does not mean a thread in a pool does only blocking operations! Instead, a thread could also run event cycle.

You may create a broad thread pool and outhand every atomic task to a closest free thread. In this case you wonʼt get delays due to earlier tasks in the same thread, but the cost of moving data between CPU caches will nearly uniformly slow all tasks down. This may be good for a broad class of applications, but wonʼt permit you to reach the really maximum performance.

About epoll: let you look at this article. Donʼt automatically treat it obsolete. What is important that it discusses difference between slow and fast clients. A slow client adds its overhead just by need to keep connection state for longer time and to switch to cold (non-cached) state. This would also add its own overhead. Migrating connections between slow-serving and fast-serving threads could be, well, painful to write, but useful for result.

To sum up: what I generally try to tell in this answer that there is no really an answer until you narrow the question. The top level task has a bunch of parameters on which you have to choose a trade-off. Iʼve added a single external reference just to point out that the manner you started to consider all this is too narrow (and the question, on the opposite, is too broad - awkward but true). But there definitely are much more resources to study. You definitely will need tunable configuration for tens of parameters and test them (and allow your server users to test and tune) on a real hardware, suggest measure techniques and typical values. You are at the very beginning of the long journey.

Answer 3

Although epoll(7) is sufficient to handle non-blocking read/write, I encountered a situation that made the model I mentioned in the question hard to implement. If there is one request per connection (socket or file descriptor), the model can still accomplish the goal (epoll for non-blocking processing requests / sending responses, thread pool for reading large static files).

However, the program becomes too complicated to handle when there are multiple requests coming from the same server. Epoll is efficient to tell me which file descriptor has something ready to be read, but it doesn't know which request if they come from the same file descriptor.

So, I switched to the other model, which uses epoll solely for monitoring file descriptors, and let thread pool handle requests (processing HTTP requests and sending responses). Each EPOLLIN event is considered as an incoming request, regardless of whether they're from the same file descriptor or not. Then, the request is constructed with some crucial fields such as associating file descriptor, server info as a work. The work is pushed to the thread pool, and the worker threads will handle request and response.