Automated Testing Using Virtual Machines

Question

I am working on making a system for the automated testing of student assignments. The biggest hurdle I am facing right now is figuring out how to sandbox the testing of the code yet still get and process the results of the test. I need to run it on a VM as there is no guarantee the code will not be malicious.

I would like to have a process similar to the following:

1. Student submits assignment
2. VM is created and is given the source code submitted by the student
3. VM runs the tests on the source code
4. VM returns results of the test back to the host operating system
5. Results are processed

Steps 2 and 4 are the ones I am looking for a good solution for. I am not sure what would be a good way automate the passing of files to the VM and the passing of results back from the VM. It seems like this is a problem that might have a solution already and I would rather not reinvent the wheel.

Answer 1

I would probably just use ssh to the VM. First copy the executable into the VM using ssh (scp or sftp), then execute it again using ssh and read it's output, possibly use ssh again to pull output files (scp/sftp again, though it's easiest to stick with standard output whenever sufficient) and shut down the VM.

The VM needs to have networking enabled for this, but you can easily prevent it from reaching outside world by configuring firewall on the host.

The VM should have it's disk image mounted in copy-on-write mode and the writable part discarded after the shutdown, so all tests run with the same initial state. You can optionally check that the copy-on-write file does not contain changes to any files it is not supposed to contain.

Answer 2

Here are a few options, in order of ascending preference:

Option 1) For 2), you can copy the file to some shared location defined on the VM. An artifact of 3) is that it can produce some output file. For 4), the host OS can poll the expected output file on the VM to know when the tests are done.

Option 2) Create some trivial web service (i.e. SOAP or HTTP API) on the VM which the host OS can call, the VM then executes, and returns response back to the web service client on the host OS. If the execution can take a long time, you have to consider setting a sufficiently high timeout on the web service client.

Option 3) If possible (not sure because I dont know what VM software you are using) use some API provided by the VM server to execute a command and get its response.

Answer 3

Is there a reason you need to spin up and spin down the VM between tests?

Personally, I would have a single VM that is capable of performing the tests and has access to a shared drive on the host machine where the tests reside. You can then iterate over the tests within a single VM by accessing the shared space.

I am only verse in VirtualBox, but you can create shared folders with relative ease, and export your VM when you have it all set up nicely so that if you come across something malicious, you just blast the broken VM and take another export (copy) of the backed up version.

The downside of this is that you don't have the results automatically given to the host from the guest, but I don't know any method of doing this on a program level anyway. Of course, the guest can write the results to the same (or different) shared folder it has access to on the host machine.