Developing a virtual machine / sandbox

Question

I'm interested in learning how a virtual machine/sandbox actually works. I have developed an 8051 emulator and also wrote a dissassembler for x86, so this part of a virtual machine is not really the problem. What I'm interested in, is the sandbox functionality of it. To illustrate what I mean consider this example.

Let's assume I have a function which simply opens a file. So nothing fancy.

 int fd = open(path);

Now when this code is executed natively it will go to the operating system and opens the file (assuming that it exists). Now when I run this in a virtual machine environment, the specified path is not the one that the operating system sees, but rather something that the VM will substitute, thus redirecting the open call. So what I'm interested in is, how a VM can do this, when the executed code is run natively (like x86 on a x86) because for an interpreted VM it is rather obvious.

When I google for virtual machines I either find only links talking about interpreters like Java, LLVM or similar, but nothing that goes into more detail. I downloaded the sourcecode from Oracle Virtual Box, but as this is a rather big codebase, it's quite hard to understand the concept just form digging in that code.

Answer 1

Now when I run this in a virtual machine environment, the specified path is not the one that the operating system sees, but rather something that the VM will substitute

This is not true for the vast majority of VMs (VMWare, VirtualBox, QEmu, Parallels, VirtualPC, …) These are all hardware virtualizers, they don't virtualize the OS. So, the open call will simply go to the OS inside the VM, which (typically) doesn't even know that it is running inside a VM. (There are performance advantages if the OS knows that it is running inside of a VM and talks to the VM instead of talking to the virtualized or emulated hardware, but that is technically no longer virtualization, it is paravirtualization.)

Of course, this just pushes the question one layer down: when the OS kernel wants to write to some specific block on the hard disk, how does that work? Well, in the worst case, the VM has to intercept the BIOS calls and map the block numbers to some file on the host system's hard disk. But this is where paravirtualization comes in: no modern OS uses the BIOS anymore, they use specialized drivers. And you could then simply provide a "virtual" driver that knows how to talk to the VM directly.

You still need to map the block numbers, of course. The simplest solution is to create a file in the host system that is the same size as the emulated hard disk. However, since hard disks tend to be mostly empty, this would be a waste of space, so most VMs employ a lazy scheme where the file only grows when needed. They might also use compression.

What you are talking about is more like OS virtualization (aka containers) than hardware virtualization. But really, it works the same way: instead of virtualizing the x86 instruction set, it virtualizes the "Linux Kernel ABI Instruction Set". IOW: it simply treats the Linux Kernel as an interpreter for a certain language (the syscalls) and then puts another interpreter on top. That's how OpenVZ works on Linux, for example, or Solaris Zones, or DLPAR on the IBM PowerSeries.

Answer 2

So the question is how to write a sandbox, in which a program can run and do no harm. I shall assume Windows, and no cooperation from the target program. I know of two ways, and they're both pretty hard.

One is to provide an environment in which every system call is emulated. Every Windows executable loads DLLs and calls entry points in them, so you have to provide your own DLLs for the program to load (via your own LoadLibrary). For each entry point you either pass it through, or block it, or modify it somehow.

The other is to rely on a suitable 'hook' feature built into the O/S. Vista and later have security features on many system APIs. You can write a filter driver to control access to disk. You can intercept the lowest level of system call and redirect those. The possibilities are endless, but rest assured, every one of them is a lot of hard work.

The typical O/S VM (VMware, HyperV) work at the driver level because emulating devices and disks is a necessary part of doing what they do.

This is all rather generic stuff. I don't know of any resources to quote, but you could certainly take a close look at Sandboxie, Chrome native client, or one of the others listed here: http://downloadsquad.switched.com/2009/03/11/five-sandboxing-apps-to-protect-your-windows-computer/.