9

I was trying to optimize my Linux server to handle 10,000 threads per process while it does just 382 right now. As per this article the following formula is used to find out total possible threads:

number of threads = total virtual memory / (stack size*1024*1024)

This means threads store all their data in virtual memory. And to the best of my knowledge, virtual memory is swap space in a Linux machine which is stored on harddisk than RAM or cache.

So my question is does our threads uses harddisk to store for processing/store its data.

If yes, then doesnt this effect performance? Can we enhance the performance by putting in them on RAM or cache? How?

If no, how exactly do threads works?

Update:

According to useless's answer, virtual memory is a system comprising roughly:

  • physical memory (RAM)
  • any swapfiles you have attached
  • hardware support for translating virtual to physical addresses and issuing page faults when a virtual address isn't available in physical memory
  • (kernel) software support for: managing the lookup tables used by that hardware handling those page faults by pulling pages in from swap on demand

Thus, Everything that is on virtual memory is collectively on RAM(Real Memory) and Hard Disk(Swap Files). And as James explain in his answer decision on Ram vs HDD is taken by Kernel using algorithims such as LRU.

  • 2
    unless your server has 10,000 CPU/Cores you are wasting your time. – Jarrod Roberson Feb 10 '12 at 14:33
  • @JarrodRoberson: Any why is that? – dragosrsupercool Feb 10 '12 at 15:07
  • 3
    10,000 threads isn't a good way to make things scale, it is a good way to make a server come to a crawl, more than 1 thread per CPU or Core is just going to make the server context switch and run slower not faster. – Jarrod Roberson Feb 10 '12 at 15:45
  • Specifically, when you say "trying to optimize my Linux server" - what are you trying to optimize? If it's throughput, then one thread per CPU with multiplexing and non-blocking I/O is likely to be better. – Useless Feb 10 '12 at 19:29
12

to the best of my knowledge, virtual memory is swap space in a Linux machine

Nope, virtual memory is a system comprising roughly:

  • physical memory (RAM)
  • any swapfiles you have attached
  • hardware support for translating virtual to physical addresses and issuing page faults when a virtual address isn't available in physical memory
  • (kernel) software support for:
    • managing the lookup tables used by that hardware
    • handling those page faults by pulling pages in from swap on demand

It's up to the kernel to make sure the virtual memory you want is cached into RAM when you want it - unless you're writing your own userspace VM layer (such as databases often do, iiuc), just don't worry about it.

  • Ok so my virtual memory assumption was wrong. Anyways a quick followup question.. Would fully loaded max threads performance be affected if SWAP Space is more than RAM? – dragosrsupercool Feb 10 '12 at 13:18
  • @dragosrsupercool: your swap space is always going to be bigger than physical memory, otherwise there's mo need to use virtual memory. – Bryan Oakley Feb 10 '12 at 14:40
  • 1
    @BryanOakley: That isn't necessarily true. Some OS's allocate a swap page for every virtual page allocated (i.e. swap must be at least as large as physical). Other OS's allocate a swap page only when there is a need to move a page out of physical memory (i.e. swap may be less than physical). The advantage of the former is that if the allocation succeeds, then swapping out that memory is always successful. The advantage of the latter is that you don't need to pessimistically allocate huge swap files to account for relatively rare situations. – mcmcc Feb 10 '12 at 15:03
  • 1
    @dragosrsupercool, performance won't be affected by the amount of RAM, swap or the ratio between them unless you're low on RAM and actually paging. sar can tell you about paging activity iirc (checked: sar -B on Linux). – Useless Feb 10 '12 at 15:50
  • @Useless: I wish to increase the number of threads till I completely utilize RAM and don't start paging. – dragosrsupercool Feb 10 '12 at 16:04
14

If the thread is actually running then the current instruction , and, any variables the thread is using must be in physical memory.

Most (in fact nearly all) programs reside in virtual memory, and, most programs use virtual memory for storage of variables.

Virtual addresses organized into chunks called pages (these are usually 4096 or 8192 byte blocks).

At any given time each block of virtual memory is stored somewhere in real memory or on the disk in the "swap space" reserved for this.

Your program code deals with virtual addresses, when you branch to a virtual address, or, request access to storage at a virtual address the system (usually at hardware level) locates the current location of the address request and maps it to your virtual address, if the address currently resides on the disk it pages it into real memory and then maps the address.

Obviously when all physical memory is in use if something is paged in then something else must be paged out, so the system looks for the "Least Recently Used" page and copies this out to disk before copying the page you requested in.

In modern systems there are several optimizations and tricks associated with virtual storage.

  • Addresses are mapped on a "per process" basis so for instance all C programs in a Linux box start the "main" process at the same address.
  • This can enable several 32 bit processes to occupy and use much more than 4GB on a machine as a 32 bit virtual address can be mapped to a real 64 bit address.
  • When processes end or memory is otherwise "free"ed the the system just marks the pages as free, they are never copied back to the swap disk.
  • Similarly when a new block of storage is requested the system just grabs a free page in real memory, no, disk IO takes place.
  • The sleep and hibernate functions force all the memory to be copied to the swap space so that all current processes and there current memory contents can be recreated on wake-up.
  • 3
    The "all C programs in a Linux box start [main] at the same address" would seem to not take address space layout randomization into account. That's used more and more today to thwart various stack smashing attack schemes. Good answer otherwise, so +1. – a CVn Feb 10 '12 at 12:53
7

First of all, you need to read more on computer memory, because you seems to lack the knowledge in that field.

A thread of execution is the smallest unit of processing that can be scheduled by an operating system. The implementation of threads and processes differs from one operating system to another, but in most cases, a thread is contained inside a process. Multiple threads can exist within the same process and share resources such as memory, while different processes do not share these resources.

So, threads are going to use available memory - whatever kind of it is available. How many threads you can start depends on the memory size and how much memory is needed per thread. If thread uses heap (not only stack), then it needs more memory, and in that case you can start less threads.

  • @VJonvic: +1 for basic thread explanation. – dragosrsupercool Feb 10 '12 at 13:33
5

The simple answer to your question is, they use virtual memory. everything uses virtual memory except a handful of processes related to the OS.

On the other hand, when your thread (or any thread, in any process) is actually running, it is using physical memory. The memory pages associated with that process are swapped in to physical memory which is where the processor does its work.

3

Virtual memory is your RAM plus your swap space. Virtual just means the address your program sees is different than the address the RAM chip sees. If you need to access memory in swap, the OS will move it into RAM first. If you don't want any swapping, just disable it. If you have enough RAM you don't really need it.

That being said, unless you have a 10,000 core processor, increasing to 10,000 threads isn't really an "optimization." Once you have enough threads to consume all the cores, plus a spare or two for when those threads are blocked, adding more threads decreases performance due to the switching overhead and cache misses. You might still want to use more threads if it makes your program logic simpler, but you will be trading off performance.

  • Yes, 10,000 is too much as my server is a 32bit single core machine. Actually, the threads are not total cpu thing. They are crawler threads, so they would be like waiting for server response sometimes. I aim to make sure cpu is totally occupied but not overload or underload. But I still fail to understand how can I know if CPU is like free or totally occupied. Is there any tool or command? – dragosrsupercool Feb 10 '12 at 16:09
  • I think you can get that information from the top command. – Karl Bielefeldt Feb 10 '12 at 16:19
  • @KarlBieledeldt: yes that was exactly what I was seeking for.. One more followup question: I just came with an idea for crawling that if some how one thread can send request for urls while the other thread receives the server response then I can keep CPU utilization high without using too many threads. Is that possible? Like sending request from one thread while receiving the response on the other thread? – dragosrsupercool Feb 10 '12 at 16:24
2

optimize my Linux server to handle 10,000 threads per process

As others explained, this is generally wrong. A thread is a costly resource, notably because it has its own call stack (typically, a megabyte) and because it is a task schedulable by the kernel. Threads are even more costly than opened file descriptors.

Read Operating Systems: Three Easy Pieces (freely downloadable textbook).

As a rule of thumb, you don't want to have many threads, and certainly not many runnable threads. The number of runnable threads should generally be at most the number of cores (or a small multiple of that), so about a dozen at most. The number of threads in a process could be slightly bigger. So unless you have a very expansive server (with many processor sockets and cores), you don't want to have more than a dozen runnable threads, and a hundred threads (most of them being idle) in your process (on your desktop).

On Linux, threads and processes are very similar (since both can be created by clone(2)) and both are tasks scheduled by the kernel. Actually the kernel scheduler is scheduling tasks which can be threads inside some multi-threaded process, or the single main thread of a single-threaded process (in that case, you'll name "process" that single thread), or kernel threads. You probably don't want to have more than a thousand schedulable tasks in total on your desktop system.

On Linux, a process is simply a group of threads sharing the same virtual address space (and sharing some other things, such as file descriptor table, etc...). Some processes have only one thread.

A virtual address space is defined by Wikipedia as

"the set of ranges of virtual addresses that an operating system makes available to a process"

(but see also this answer explaining that the terminology is not universal, and some Microsoft documentation uses a different and incompatible definition).

On Linux, proc(5) is useful to understand the virtual address space of some processes. Try both
cat /proc/self/maps and cat /proc/$$/maps in a terminal. See also this, and pmap(1) & ps(1) & top(1).

All user-space programs are running in some process and using virtual memory so every process has its own virtual address space. The physical RAM is a resource managed by the Linux kernel, and applications don't have direct access to RAM (except by mmap(2)-ing /dev/mem, see mem(4)).

So a process don't use directly RAM. It uses virtual memory and has its own virtual address space. The kernel uses paging to manage physical RAM pages and provide the virtual address space and the process abstractions. At any time (even when your process is idle, or when it is running) the kernel could page out some pages (e.g. swap them on the disk). The kernel is configuring the MMU (and handling page miss hardware exceptions in some interrupt handler, either by fetching the page from disk or by propagating a segmentation fault to the process, see signal(7))

You could have green threads above system threads (but green thread libraries are difficult to implement and debug). Look into goroutines used in Go for a fancy example. See also setcontext(3).

Sometimes, your system may experiment thrashing. This happens when the total virtual memory (needed by all processes) exceeds -by a large factor- the available physical RAM. Then your computer becomes unresponsive. Read about resident set size, demand paging, working set, memory overcommitmment, ASLR.

See also -for Linux- fork(2), clone(2), mmap(2), madvise(2), posix_fadvise(2), mlock(2), execve(2), credentials(7), pthreads(7), futex(7), capabilities(7).

protected by gnat Apr 6 '18 at 13:38

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.