My application does a lot of database inserts, so disk I/O is a big part of the workload. QA does almost all testing on VM's. I'm concerned that tests intended to detect performance regressions won't give valid repeatable results in a VM environment, since other activity on the physical machine will affect the application performance.
Is this a legitimate concern, or do modern virtual environments have a way of truly isolating an application's environment which would allow for repeatable performance tests?
I want to put my application and database on a "machine", run a test and note how much time it took (which will be some number of hours/minutes, not seconds/milliseconds). Later in my development cycle, I want to run the same test and check whether performance has regressed due to code changes. When running on a dedicated physical machine, I get reasonably consistent results. My question is, if I run this test on a virtual machine, might I get significant differences in run time due to work being done by other VM's on the same physical box? Is there any way to configure the VM to control for this, considering that disk I/O is a major part of the workload?