0

I'm building an HTTP server in C using epoll and pthread. The idea is to have an event loop in the main thread monitoring file descriptor and a thread pool to handle computationally-expensive operations.

What I'm struggling is that what type of operation should be done by an event loop and by a thread pool. Here is what I expect to be done:

  • My epoll event structure will store a request struct and a response struct.
  • After adding new file descriptors to monitor (EPOLLIN), the server will read from a file descriptor if there is a new request, and store to the request field in the epoll event
  • After reading the request, the event loop will push it to a queue. A thread pool will take requests from the queue and process all the requests (assuming all static files).
  • After processing, the thread will store a response back and switch the event flag to EPOLLOUT. The event loop will monitor and send out the response.

Here is what I've seen in many examples:

  • Accept a new connection and add the file descriptor to the epoll interface.
  • If there is an event, except from accepting new connections, the thread pool will handle data from the file descriptor.

In a nutshell, I think that threads should only handle operations such as reading large static files, and the event loop should handle network and socket operations such reading from or writing to sockets. What do you think?


(I updated this post with my own solution)

4
  • it's probably most convenient for you if the main thread does everything except the things that are too slow for it to do. What's the point of using a "thread-less" design to read the request if you're going to use a thread to send the response anyway? Every request has a response, so every request is going to need a thread.
    – user253751
    Jul 22, 2023 at 16:14
  • Based on your comment, does it mean that non-blocking socket read/write operations can be done in the main thread, and expensive tasks like reading static files should be done by threads? Jul 22, 2023 at 16:42
  • 1
    It is ultimately up to you. I strongly recommend that you do not mix up client sockets between threads. If a second thread is using the socket, the main thread should have completely forgotten about it (as much as possible) and that means it should not be registered for epoll at all. If you don't like that, then the socket can be the main thread's problem which means the second thread should not use it at all.
    – user253751
    Jul 24, 2023 at 11:08
  • If you try schemes like having the main thread epoll the socket at the same time the second thread is writing, maybe you can make it work, but I guarantee it will have some very complicated bugs that only happen sometimes. It will never work reliably unless you put a ton of thought into it. This always ends up happening when you try to use something like a socket on more than one thread at the same time and you don't have a PhD in multithreading.
    – user253751
    Jul 24, 2023 at 11:09

3 Answers 3

2

epoll only notifies you that an specific fd is able to receive data, or has data to read. then reading from non-blocking socket or writing into it doesnt block your execution or waits til data is transfered.

reading in blocking IO result in: if data is available your buffer is filled with data, otherwise thread goes to sleep til data is available or a failure detected

writing in blocking IO result in: if able to write, then you write and thread goes to sleep til data is written, otherwise thread goes to sleep til able to write

but in non-blocking IO, thread never goes to sleep, you only receive notification of availibility for reading and writing for specific fd. and for detecting "its enough to read or write", read or write will return -1 and errno is set to EAGAIN or EWOULDBLOCK.

dont worry about errno, its per thread, your execution wont mess.

reading or writing is the same for kenel in any situation, no matter its blocking or non-blocking. its all about reducing number of context switiching by reducing number of threads. hence you receive more performance.

when you are in a situation you need to do any async task, you need more playing with states than doing traditional jobs.

for example:

in a blocking IO for parsing http requests you read as much as a full http header is parsed then step into next phase. parsing in blocking IO is easy, because you dont miss which part of your parsing phase you were, as you dont leave that part.

but the game in non-blocking IO is different, because everytime you receive a partial data, consider receiving data as below:

first

GET /

then

index.php http/1.1

then

\r\n\r\n

so how do you parse that data?

you have two choices:

  1. store as much in a cache til you receive double \r\n then parse with your previous algorithm

  2. start parsing from a previous stored state, if done return, if not, store state for next data arrival ...

receiving data in partial as above is a rare case, but in busy networks with some custom configs you will face such a case.

but the problem is not parsing headers, its all about you have to do everything in partial as your data is ready partially. so you need to store states of the place you previously were in, for any connection.

so when you are doing non-blocking IO, no matter you are doing all things in one thread or pass to other threads to do stuffs, anyway you are lost in lots of partially done stuffs.

so you need to properly store states of anything, yes anything. and continue from that state. and that makes your code more complicated as the problem is complicated.

we wait for another nginx with more performance from you ;)

4
  • I actually used your first choice: read as much data as possible into a temporary buffer and continue to parse when encountering \r\n\r\n (assume no POST/PUT requests). This is all done within epoll. I also use this strategy for response. My problem is when I fire multiple requests from the same client file descriptor, I couldn't find a way to handle that. I think it might be the problem of HTTP pipeline. Btw, I would still recommend people to use Nginx over mine. :) Oct 15, 2023 at 19:02
  • @RichardH.Nguyen what do you mean by "I fire multiple requests from the same client file descriptor"? do you mean keep alive feature? Oct 16, 2023 at 1:35
  • No, it's not about keep-alive feature. I wrote a simple client program that has one active socket and use multiple threads to send requests. I couldn't handle that. I also tested with Firefox. When Firefox wants to make multiple concurrent requests, it opens one file descriptor per request. This I guess it's due to a problem known as HTTP pipeline in HTTP/1.1. Oct 16, 2023 at 2:20
  • @RichardH.Nguyen each connection can only be used for one request-response pair, then you can reuse that socket. please update your question with more details and semi code hope I can help Oct 16, 2023 at 13:14
1

It seems you are trying now to find a simple answer to a complex multi-axis question under conditions that nobody knows except you. We donʼt know how exactly you organize network interaction - for example, how many copyings are used there (they are CPU-bound part of a network server, especially for a text one like HTTP). We donʼt know how your inter-thread interaction is organized. There would be loads of other factors on which a solution really depends.

Well, there are some principal considerations which should affect the design.

As you mentioned, disk reading is, with default functions or mmap, blocking. If you use it that way, a thread pool is a solution. But, if you mentioned epoll, this definitely means Linux. Having Linux you may utilize io_uring. This is a rather new tecnhnique, but seems already attaining the stable state. But even it the cost to parse requests, generate response headers, call handlers for URL parsers, find file in FS, etc. - will be mainly unsplittable into small asynchronous tasks.

You said - to thread pool for large static files? Well, it seems you mean that cost of a task offloading to a pool could be greater than reading a single file. But you may fall into IO overload. Blocking of the main thread with a delay of IO operation that was postponed for even a few tens of milliseconds due to scheduler's insane decision... seems this is not what Iʼd expect from such a server. Better is to decide that any new connection goes to a thread pool, but - it is crucial - it does not mean a thread in a pool does only blocking operations! Instead, a thread could also run event cycle.

You may create a broad thread pool and outhand every atomic task to a closest free thread. In this case you wonʼt get delays due to earlier tasks in the same thread, but the cost of moving data between CPU caches will nearly uniformly slow all tasks down. This may be good for a broad class of applications, but wonʼt permit you to reach the really maximum performance.

About epoll: let you look at this article. Donʼt automatically treat it obsolete. What is important that it discusses difference between slow and fast clients. A slow client adds its overhead just by need to keep connection state for longer time and to switch to cold (non-cached) state. This would also add its own overhead. Migrating connections between slow-serving and fast-serving threads could be, well, painful to write, but useful for result.

To sum up: what I generally try to tell in this answer that there is no really an answer until you narrow the question. The top level task has a bunch of parameters on which you have to choose a trade-off. Iʼve added a single external reference just to point out that the manner you started to consider all this is too narrow (and the question, on the opposite, is too broad - awkward but true). But there definitely are much more resources to study. You definitely will need tunable configuration for tens of parameters and test them (and allow your server users to test and tune) on a real hardware, suggest measure techniques and typical values. You are at the very beginning of the long journey.

2
  • Thanks for your thorough response. I'm not sure how to narrow my question more since that was very particular, at least to me. I use the NodeJS model: <nodejs.org/en/docs/guides/dont-block-the-event-loop />. It says that "Node.js runs JavaScript code in the Event Loop (initialization and callbacks), and offers a Worker Pool to handle expensive tasks like file I/O". However, all examples I could find all the internet would have an event loop just for accepting and monitoring, and reading/writing goes to thread pools. I wonder how people design such a program. Jul 31, 2023 at 22:17
  • @RichardH.Nguyen NodeJS design (as well as e.g. Python, Perl and many others) is fast and clear but it was not designed to carry really high load. So it isn't surprising it isn't for wide scaling. After invention of what you described, newer NodeJS applies separate processes and easy object exchanging between processes. This is how they handle when CPU gets the narrow place.
    – Netch
    Aug 2, 2023 at 5:30
1

Although epoll(7) is sufficient to handle non-blocking read/write, I encountered a situation that made the model I mentioned in the question hard to implement. If there is one request per connection (socket or file descriptor), the model can still accomplish the goal (epoll for non-blocking processing requests / sending responses, thread pool for reading large static files).

However, the program becomes too complicated to handle when there are multiple requests coming from the same server. Epoll is efficient to tell me which file descriptor has something ready to be read, but it doesn't know which request if they come from the same file descriptor.

So, I switched to the other model, which uses epoll solely for monitoring file descriptors, and let thread pool handle requests (processing HTTP requests and sending responses). Each EPOLLIN event is considered as an incoming request, regardless of whether they're from the same file descriptor or not. Then, the request is constructed with some crucial fields such as associating file descriptor, server info as a work. The work is pushed to the thread pool, and the worker threads will handle request and response.

2
  • if each epollin event is a request, dont you run into problems when a request is only partially transfered? Like you dont have a gurantee that a single read will read the entire request data.
    – The Fool
    Oct 29, 2023 at 19:46
  • @TheFool I did run into the partial read/write problem on epollin/epollout events respectively due to non-blocking mode. My attempt to solve this problem is to provide two temporary buffers over a connection struct. epollin events will read as much as possible into the receiving buffer. If the read system call returns 0, it marks the end of the request and passes the buffer into thread pool for processing. Same thing for the sending buffer, a thread will construct a response message and put it into the sending buffer. The write system call writes as much as possible until the offset = len Nov 1, 2023 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.