0

I'm studying asynchronous IO, concurrent models for IO and how things works on windows, linux and most used web frameworks.

I'm struggling on understanding why single-threaded event loops like the one used by node.js or ngnix, in case of handling requests that wants to make an IO operation (let's say an HTTP request to get data from another service), uses a dedicated thread for each IO operation, instead of using just one thread and Epoll to handle all of them.

Let me explain better:

  • Every time I read about Epoll I read examples or tutorial from a server perspective: I'm a server -> I hate thread's context switch -> Epoll let me handle a lot of incoming requests with just one thread. Nice.
  • So I build a server tech like node.js: single-threaded event loop and non-blocking IO operations. How can I make non-blocking IO operations? I execute them on dedicated IO-threads and each of this thread will block until the IO operation is completed (letting the main event thread free to handle other requests).
  • It means that if I receive 100 requests, and for each of them I have to make an IO operation, I could end up using 100 threads or the maximum number of threads available in the thread pool leaving lot of IO operation waiting for a free thread.

So what's the point of using Epoll or others multiplexing libs to save on thread count for managing incoming requests if I could end up doing a lot of context switches for managing IO operations needed by each handled request to complete? Why are not these IO operations "batched" to be performed on a single (or few) threads using Epoll?

Improve this question

1 Answer 1

Reset to default
2

Select and Epoll were and are fundamentally flawed APIs. They tell us efficiently if I/O operations are possible on a particular file descriptor, but do not actually let us perform the I/O operation in an async manner: once you perform a read or write, you're blocked until that operation is done.

Thus, the common workaround is to delegate potentially-blocking I/O operations to a thread pool (NOT one thread per operation). This is reasonably efficient, since those threads will spend most of their time blocked on I/O operations (either on syscalls or waiting for new I/O tasks from the main thread). Sending a task to the I/O thread pool doesn't require a context switch. The number of open connections can be much higher than the number of I/O operations that are currently being performed.

Better approaches are available, notably completion ports as pioneered by Windows NT. Linux finally got fully async I/O with io_uring in version 5.1 (May 2019).

Improve this answer
5
  • Thanks for your answer! If possible, I ask some clarification: (1) I know there is a thread pool to handle IO, but how each operation is delegated to the thread pool? If I enqueue 10 IO operations and I have 4 IO threads free in the pool I end up with 4 operations running each on a thread and 6 waiting for a free thread, right? Or they are somehow "all distributed" among the 4 threads? (2) Why you say there is no context switch? On a 4 core CPU, I can have one core handling the main loop and the other three handling IO threads, won't the fourth IO thread never gain its turn on the CPU?
    – Mattias Santoro
    Commented Apr 27, 2021 at 14:00
  • @MattiasSantoro (1) The point of the thread pool is that those threads are blocked, not the main thread. But if you have 10 tasks for 4 threads then yes, 6 will have to wait. (2) The main thread doesn't have to context-switch (syscall or to another thread) to enqueue IO tasks: the queue can use shared memory. The IO threads will context switch a lot. Since the IO threads spend most of the time waiting for something, they're not hogging their own CPU each – threads that are blocked will not be scheduled, and the scheduler can run other tasks meanwhile.
    – amon
    Commented Apr 27, 2021 at 14:40
  • Select and epoll can be made to work if you also use non-blocking I/O. You don't need a thread pool.
    – Criticizing Israel not allowed
    Commented Jul 26, 2022 at 12:45
  • @user253751 Really depends on how non-blocking you want to be. Select/epoll only tell you whether data is available, but you'd still have to use blocking syscalls like read/write/recv/send to actually perform the IO. Posix AIO exists but no one uses it since it isn't actually asynchronous in many situations. Only io_uring provides for truly async IO on Linux.
    – amon
    Commented Jul 26, 2022 at 13:23
  • 1
    @amon If a socket is set to non-blocking mode, then read, write, recv and send do not block. That's why it's called non-blocking mode.
    – Criticizing Israel not allowed
    Commented Jul 26, 2022 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.