My question will be mostly about Linux and contemporary X86 hardware.
Clarifying the terms
async event based programming: spawning fixed amount of threads and using some user space scheduling mechanism. Usually callbacks are handled in a continuation style, so the callback will be executed in the thread of the consumer, not producer.
thread, blocking based programming: using classic OS threads and mutexes and semaphores. Waiting is handled by blocking the calling thread. Although not usually used, in my perspective this requires cooperative threads to be effective.
The status quo from my POV
It seems like more and more people turn away from the classic way of using syncrhonization/cooperation primitives for threading and from blocking (as in making current thread sleep) in general. Their problems seems to be mainly based on the following (italics are my thoughts on the matter):
Blocking synchronization is prone to deadlocks (async code can livelock too)
Too many threads cause oversubscription and OS scheduler makes latency sky high (fixable?)
Whatever libraries that use threading do not provide any way of cooperation, thus causing the problem above
High cost of a thread spin-up (perceived?)
Not usable if sleep/wakeup is noticeable compared to active time (true)
#5 is not fixable without patching kernel and #3 could be fixed by better teaching.
The can of worms
Async systems usually rely on events that trigger something else, thus cascading. Even if the above problems were fixed by async event based programming, it creates harder issues that are solvable by blocking thread based programming:
Starvation (very hard to achieve fairness without OS support, which in turn utilizes hardware clocks to enforce CPU time slices)
Can break when only one worker is allocated (this is mostly true of lock free queues, nothing can cause progress condition when the worker is not giving CPU time to another task, although this is solved with coroutines)
Exclusivity and ABA problems (with OS threads no thread can be scheduled on multiple CPUs at the same time, while with event based programming some event might be fired more than needed)
Complexity either goes into the library (which, if written with poor interface, pretty much doubles the mental complexity) or into the user code
Why reinvent the wheel?
For most complaints I have seen with thread based blocking programming, I couldn't find any reason why it couldn't be solved with classic approach. The only problem is when a thread needs to run for so small amount of time that typical ~10 microsecond delay will start becoming noticable. It is usually I/O bound applications. On all other cases I found that non-blocking code is usually less efficient on the normal path (non contending), because it does more operations to ensure atomicity. It seems like almost all problems could be solved by using blocking in a better way (don't hold lock for more than very few operations) and exposing parallelism instead of just spawning threads (algorithms could use task queues, for example). Real-time is not possible without preemption, so async event driven code is not real-time by definition. Even the coroutine based frameworks pretty much duplicate the OS scheduler and synchronization primitives.
Examples when threads are superior
On one occasion I had to utilize only 2 CPU cores for a stream of tasks, where some tasks take about 100 times longer than the others. Having two of those would pretty much stop anything from making progress. I did the following:
Allocate two threads for lightweight tasks and make the producers put lightweight tasks into a queue
Spin up a thread for every heavy task
Put everything under
SCHED_RRwith a timeslice set to upperbound of a lightweight task, so that even in case of heavier tasks, lighter ones could make progress
For async event based programming, this would require support from the callee, and it would have predefined amount of preemption points, while OS based one is a pretty much coroutine with as many preemption points as possible, albeit with less control over where exactly it will be preempted.
Since async event driven programming offers so little benefits, why is there shift towards it? What am I missing?