Why do some VMs manage their own memory instead of relying fully on the system allocator?

Question

As far as I know, most "serious" VM implementations, such as CPython and the Oracle JVM, do not request new memory from the operating system ("malloc()") each time they create a new object.

As far as I understand it, these VMs usually request a large chunk of memory once, and then manage it internally for allocating objects as the program is running. (Pretty sure Oracle JVM requests all memory once at startup, and CPython requests blocks of memory as it goes).

I'd like to understand - how is this approach better / more performant than simply requesting memory from the OS ("malloc()"?) each time the VM needs it?

As a more specific question - is code B more performant than code A? Why?

Code A:

Thing* allocate_thing(void) {
    Thing* thing = malloc(sizeof(Thing));
    thing_init(thing);
    return thing;
}

Code B:

static uint8_t* memory_buffer = NULL;
static int bytes_offset = 0;

void init_memory_buffer(void) {
    /* called once at beginning of program */
    memory_buffer = malloc(1024 * 1024);
}

Thing* allocate_thing(void) {
    Thing* thing = (Thing*) &memory_buffer[bytes_offset];
    thing_init(thing);
    bytes_offset += sizeof(Thing);
    return thing;
}

Answer 1

One reason is that applications, being higher level, know their requirements better than the underlying system.  For example, if the application is doing garbage collection, that benefits greatly from a custom algorithm.

As a more specific question - is code B more performant than code A? Why?

Typically, yes, — advancing a high water mark to separate in-use memory vs. free memory — would be more efficient than malloc.  There are many versions of malloc and most of them search (e.g. free lists) a bit in some manner in order to allocate a memory block.  (To be clear, though, your code does over simplify the allocation operation in a garbage collected system.)

malloc-style memory allocation somewhat splits memory management costs between allocation and free'ing.  Some work is when free is invoked, but of course, the size of the next allocation request is not known until the malloc call happens, so some work happens there, too.  Further, using malloc/free, memory (depending on the application) is constantly being returned to the free pool (via free, e.g. in small amounts), and one thing that mallocs may try to do is reuse returned memory before carving up (or requesting from the operating system) larger blocks.  (This can be cache friendly, as a recently freed block is probably more likely to be hot in the cache.)

A garbage collector, by contrast, requires more effort to reclaim unused memory (in part b/c the application is not constantly releasing objects as there is no manually invoked free operation), so does this reclamation operation much less often.  This means that it tends to use a simpler and faster allocation, though due to the expense of reclamation (despite being done less often) the total expense of this kind of memory management can outweigh that of malloc/free.

Of course, we can construct a specific workload that shows one being better than the other and vice versa.

Answer 2

malloc is actually calling once in a while the system allocator (e.g. mmap on Linux).

And most VM have more precise knowledge about their allocation pattern. So they could do things better.

For example, a Lisp VM may know that most allocations are cons cells.

Of course, clever garbage collections algorithms may want or need to seggregate allocations. Read the GC handbook for more (e.g. think of generational GCs).

Answer 3

There is a JVM which does this: the Azul JVM for the Azul JCA platform.

On this JVM, the memory for each object is allocated using the Operating System in a separate page with a separate page table entry. However, this is a special Operating System that was specifically designed for this JVM, and a special CPU that was specifically designed for this JVM.

When Intel introduced Nested Page Tables, Azul realized that they could use the same trick on AMD64 now.

Azul tried to port this JVM onto Windows and Linux, but they found that the Virtual Memory Subsystems of both Windows and Linux simply could not handle the workload and would either slow to a crawl or crash completely. They made a very crude patch for Linux which basically replaces large chunks of Linux's Virtual Memory Subsystem with their own to prove that it can work in principle. However, this patch was never intended to be merged into Linux.

Nowadays, the Azul Zing JVM does actually run on PCs, but it actually runs without an Operating System directly on a hypervisor side-by-side with the "host" OS.

So, the answer is: the reason VMs don't do this, is because it has been tried and it doesn't work, unless Operating Systems significantly change how their Virtual Memory Subsystems work. The fact is that VMSs of modern OSs are highly optimized for C-like languages and actively hostile to Garbage-Collected implementations, so there is not much sense in using them.

Answer 4

I'd like to understand - how is this approach better / more performant than simply requesting memory from the OS ("malloc()"?) each time the VM needs it?

As has been pointed out in comments and other answers, most implementations of malloc(3) don't call the system each time. (That's why the manual lists it as a library function in section 3 rather than a system call in section 2.) A naive implementation of malloc() could simply adjust the break value, each time someone calls it. With brk(2) being a system call and requiring a switch from user to kernel and another switch back, that gets really expensive really quickly. It also doesn't cover how to deal with free(2).

Back when CPU cycles weren't as cheap or plentiful as they are now, the extra expense had a pronounced effect on program performance. Someone figured out that it was much cheaper to adjust the break by a larger amount once and dice up the memory into bits by twiddling the pointers in a doubly-linked list. Since everything in software is a trade-off, doing that gets speed at the expense of space. That approach is essentially how modern malloc() works, although there are some different things done in special situations.

Most VMs are, at heart, programs that emulate bare metal. Like a real processor, they have to provide memory space and don't know ahead of time where the programs they run will want to write. Without developing a complex virtual memory scheme (i.e., writing an OS within an OS), all they can do is allocate a large swath of memory at startup and, as OJVM does, grow it if needed.

If that sounds wasteful, on some OSes, Linux being one, it isn't. From the malloc(3) manual page on Linux:

By  default,  Linux  follows  an optimistic memory allocation strategy.
This means that when malloc() returns non-NULL there  is  no  guarantee
that  the  memory  really  is available.  In case it turns out that the
system is out of memory, one or more processes will be  killed  by  the
OOM   killer.    For   more   information,   see   the  description  of
/proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and
the Linux kernel source file Documentation/vm/overcommit-accounting.

As a more specific question - is code B more performant than code A? Why?

Now that you know how malloc() works, the answer is that they're equally-performant because neither will call out to the system very often. Your "B" implementation is doing the same thing malloc() does internally and is why it's best to use the system-provided functions: somebody's already thought of this stuff and written an implementation that's been refined over many years.

One other crucial differnce is that the "A" implementation will continue returning memory as long as the system has it available and "B" will run out after you've used up the megabyte you allocated.