I am working on making a system for the automated testing of student assignments. The biggest hurdle I am facing right now is figuring out how to sandbox the testing of the code yet still get and process the results of the test. I need to run it on a VM as there is no guarantee the code will not be malicious.

I would like to have a process similar to the following:

  1. Student submits assignment
  2. VM is created and is given the source code submitted by the student
  3. VM runs the tests on the source code
  4. VM returns results of the test back to the host operating system
  5. Results are processed

Steps 2 and 4 are the ones I am looking for a good solution for. I am not sure what would be a good way automate the passing of files to the VM and the passing of results back from the VM. It seems like this is a problem that might have a solution already and I would rather not reinvent the wheel.

  • "I need to run it on a VM as there is no guarantee the code will not be malicious." Is there any way you can take advantage of user permissions to prevent user from accessing things it shouldnt? Also you could limit the memory allotted to the program so it doesnt take up too much memory (i.e. maliciously). – jordan May 19 '14 at 1:53
  • Also, depending on your programming language/framework of choice, you may be able to use a sandbox and keep the solution all on your box. Of course, I suppose it's a potential tradeoff in security but if the VM has network access that's not exactly going to be perfect either. So here I'm thinking for .NET-esque environments either AppDomains to a lesser degree or MonoSandbox if that does it for you. Just trying to give you more options... – J Trana May 19 '14 at 2:04
  • For a lot of applications, I agree, there are easier ways to do this. But for some classes, say an OS class, the assignment requires the application to access a lot of low level/protected stuff so it would be nice to have a more generalizable solution. – Justin May 19 '14 at 5:30

I would probably just use ssh to the VM. First copy the executable into the VM using ssh (scp or sftp), then execute it again using ssh and read it's output, possibly use ssh again to pull output files (scp/sftp again, though it's easiest to stick with standard output whenever sufficient) and shut down the VM.

The VM needs to have networking enabled for this, but you can easily prevent it from reaching outside world by configuring firewall on the host.

The VM should have it's disk image mounted in copy-on-write mode and the writable part discarded after the shutdown, so all tests run with the same initial state. You can optionally check that the copy-on-write file does not contain changes to any files it is not supposed to contain.


Here are a few options, in order of ascending preference:

Option 1) For 2), you can copy the file to some shared location defined on the VM. An artifact of 3) is that it can produce some output file. For 4), the host OS can poll the expected output file on the VM to know when the tests are done.

Option 2) Create some trivial web service (i.e. SOAP or HTTP API) on the VM which the host OS can call, the VM then executes, and returns response back to the web service client on the host OS. If the execution can take a long time, you have to consider setting a sufficiently high timeout on the web service client.

Option 3) If possible (not sure because I dont know what VM software you are using) use some API provided by the VM server to execute a command and get its response.

  • I am not committed to a VM software at this point. If there were one that lent itself particularly well to this application I would probably use it. Otherwise the web service sounds like it might be the way to go. – Justin May 19 '14 at 5:31
  • Why http? Ssh sounds much easier. – Jan Hudec May 19 '14 at 11:51
  • Good point, ssh is also a great option. – jordan May 19 '14 at 12:25

Is there a reason you need to spin up and spin down the VM between tests?

Personally, I would have a single VM that is capable of performing the tests and has access to a shared drive on the host machine where the tests reside. You can then iterate over the tests within a single VM by accessing the shared space.

I am only verse in VirtualBox, but you can create shared folders with relative ease, and export your VM when you have it all set up nicely so that if you come across something malicious, you just blast the broken VM and take another export (copy) of the backed up version.

The downside of this is that you don't have the results automatically given to the host from the guest, but I don't know any method of doing this on a program level anyway. Of course, the guest can write the results to the same (or different) shared folder it has access to on the host machine.

  • 1
    If the VM is not restarted, one test could leave around some junk that would then affect the next one. – Jan Hudec May 19 '14 at 11:52
  • @JanHudec My reading of the OP is that the sandboxing is specifically for the possibility of malicious code, not because there is a worry one test may leave effects on another. Restarting a VM wouldn't clear any of the malice, you would have to re-import a baseline VM for this which seemed excessive in comparison to what the OP seemed to need. – Mitch Kent May 19 '14 at 13:04
  • I don't claim restarting is sufficient to clear the malice, but it is certainly necessary. "Re-import baseline" is however trivial operation provided that the VM uses copy-on-write and I definitely suggest doing that. – Jan Hudec May 19 '14 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.