One significant drawback with MxN threads is relative to blocking I/O.
If the system has blocking I/O, then the kernel threads (N's) can become blocked. So, assuming you have 4 processors and use N=4, your application might lose access to a processor for the duration of the blocking I/O operation.
This is not so much a property of the scheduler itself but of the interaction of the user code with the operating system.
In the following text, LWP is a light weight process equivalent to a kernel thread.
Prior to Solaris 2.6 software, the kernel used a special signal, SIGWAITING, to inform the threads library that all LWPs were blocked in the kernel. This gave the library the opportunity to create another LWP so it could to continue to run other, nonblocking threads. In Solaris 2.6 software, this mechanism was augmented by the preferential use of a “door upcall.” Essentially, this involves the kernel being able to call into the user-level thread scheduler to adjust the number of LWPs in the process’ pool of LWPs. This door mechanism is more efficient than a signal, but if necessary, Solaris 2.6 software falls back to using the SIGWAITING mechanism.
In this text, there is no mention of how to perform the respective reduction of the N threads as would be appropriate when the blocking thread becomes unblocked. Suffice it to say that this is complicated.
Problems with blocking I/O can be avoided by using always using non-blocking I/O, and asking the scheduler to run a different M thread on the now-available N thread when doing the async I/O request. A library layer might translate blocking calls into such non-blocking calls that cooperate with the MxN scheduler and the async I/O completion mechanism. Apparently Erlang does this.
The are also potential issues are around thread priority with MxN as the kernel only sees the N threads, and the M threads are process-specific. So, no one sees all the M threads together across all processes, whereas with 1x1 threads, the kernel sees all threads together and thus has the widest view to make the most balanced decisions.
I believe this is mitigated if there is only one real process (e.g. one Erlang virtual machine) on the system.
A history of the trade-offs over the years of operating system development would be interesting to digest. It looks to me like Solaris implemented MxN threads fairly early but later switched to 1x1 by lowering the cost of kernel threads.