optimize my Linux server to handle 10,000 threads per process
As others explained, this is generally wrong. A thread is a costly resource, notably because it has its own call stack (typically, a megabyte) and because it is a task schedulable by the kernel. Threads are even more costly than opened file descriptors.
Read Operating Systems: Three Easy Pieces (freely downloadable textbook).
As a rule of thumb, you don't want to have many threads, and certainly not many runnable threads. The number of runnable threads should generally be at most the number of cores (or a small multiple of that), so about a dozen at most. The number of threads in a process could be slightly bigger. So unless you have a very expansive server (with many processor sockets and cores), you don't want to have more than a dozen runnable threads, and a hundred threads (most of them being idle) in your process (on your desktop).
On Linux, threads and processes are very similar (since both can be created by clone(2)) and both are tasks scheduled by the kernel. Actually the kernel scheduler is scheduling tasks which can be threads inside some multi-threaded process, or the single main thread of a single-threaded process (in that case, you'll name "process" that single thread), or kernel threads. You probably don't want to have more than a thousand schedulable tasks in total on your desktop system.
On Linux, a process is simply a group of threads sharing the same virtual address space (and sharing some other things, such as file descriptor table, etc...). Some processes have only one thread.
A virtual address space is defined by Wikipedia as
"the set of ranges of virtual addresses that an operating system makes available to a process"
(but see also this answer explaining that the terminology is not universal, and some Microsoft documentation uses a different and incompatible definition).
On Linux, proc(5) is useful to understand the virtual address space of some processes. Try both
cat /proc/self/maps and
cat /proc/$$/maps in a terminal. See also this, and pmap(1) & ps(1) & top(1).
All user-space programs are running in some process and using virtual memory so every process has its own virtual address space. The physical RAM is a resource managed by the Linux kernel, and applications don't have direct access to RAM (except by mmap(2)-ing
/dev/mem, see mem(4)).
So a process don't use directly RAM. It uses virtual memory and has its own virtual address space. The kernel uses paging to manage physical RAM pages and provide the virtual address space and the process abstractions. At any time (even when your process is idle, or when it is running) the kernel could page out some pages (e.g. swap them on the disk). The kernel is configuring the MMU (and handling page miss hardware exceptions in some interrupt handler, either by fetching the page from disk or by propagating a segmentation fault to the process, see signal(7))
You could have green threads above system threads (but green thread libraries are difficult to implement and debug). Look into goroutines used in Go for a fancy example. See also setcontext(3).
Sometimes, your system may experiment thrashing. This happens when the total virtual memory (needed by all processes) exceeds -by a large factor- the available physical RAM. Then your computer becomes unresponsive. Read about resident set size, demand paging, working set, memory overcommitmment, ASLR.
See also -for Linux- fork(2), clone(2), mmap(2), madvise(2), posix_fadvise(2), mlock(2), execve(2), credentials(7), pthreads(7), futex(7), capabilities(7).