Are repeatable performance tests possible on a VM?

Question

My application does a lot of database inserts, so disk I/O is a big part of the workload. QA does almost all testing on VM's. I'm concerned that tests intended to detect performance regressions won't give valid repeatable results in a VM environment, since other activity on the physical machine will affect the application performance.

Is this a legitimate concern, or do modern virtual environments have a way of truly isolating an application's environment which would allow for repeatable performance tests?

I want to put my application and database on a "machine", run a test and note how much time it took (which will be some number of hours/minutes, not seconds/milliseconds). Later in my development cycle, I want to run the same test and check whether performance has regressed due to code changes. When running on a dedicated physical machine, I get reasonably consistent results. My question is, if I run this test on a virtual machine, might I get significant differences in run time due to work being done by other VM's on the same physical box? Is there any way to configure the VM to control for this, considering that disk I/O is a major part of the workload?

Answer 1

To address some of the comments first, yes all the hypervisors worth building a VM environment upon have the ability to limit resource consumption and also guarantee minimums. You can leverage these settings to establish performance service levels. Establish the minimum requirements and translate that into a VM sizing. For example, a 2 core vCpu and 4GB RAM, and allocate a VM that cannot be given anymore and validate the app performance is within the requirements.

This guide from vmWare explains from that products perspective: https://www.vmware.com/files/pdf/partners/tap/directions-vmware-ready-testing-application-software.pdf

A key point in that guide is to do your performance testing while also monitoring the hypervisor to make sure the CPU < 70% and memory is not starved so that virtual swapping is happening.

A general rule of thumb is when either of those things happen (CPU of the hypervisor > 70% or physical memory exhausted on the hypervisor) you can expect to see a degradation of performance of the VMs.

You can do the same for the storage and net IO dimensions, but this is going to be driven by the investment made in the underlying architecture. My organization puts VMs on SAN with multiple 10GB uplinks to the SAN, and that results in the VMs seeing higher storage IO than they would see with local disk.

Ideally, the organization your app will be deployed has a managed VM infrastructure with monitoring of the hypervisors being done and being proactive to manage such that the hypervisors aren't pushed to the point that creates VM slow down. If you can, get metrics on the hypervisor to understand its general performance trends, and then also track that while testing and state your app's requirements based on the assumption that the hypervisor will be managed to stay within those same parameters.

If you can't get metrics about the hypervisors resource and performance, then really all you can do is measure resource used by your OS and make that the requirement that needs to be guaranteed by the hypervisor.

Since your concern is mainly around storage IO, this blog focuses on measuring IOPs: https://blog.synology.com/?p=146

Answer 2

Much of what you are asking is highly dependent on the environment. There are vendors that allow the apportioning of min and max CPU and IO to a VM among a really large number of other options. The downside of this is that as you start chopping up the resources like this, you will generally end up with lower utilization rates. Most applications do not use CPU continually at the same rate all the time so by letting VMs use what they need from a shared pool, you tend to be able to achieve an average utilization (as a percentage of all CPU available) that is higher than if you were to allocate min and max for each.

In general my experience with performance results is that performance can vary for many reasons that can be hard to control. I think it makes more sense to think of performance in more statistical terms. If you are going to be running this application on a VM in production, I would say you should absolutely run it in a VM in performance testing and configure it the way you plan to configure it in prod. Run lots of tests at different times of the day without changing anything and see what variance you get in results. This, by itself is useful info. You may find that you want to reconsider your configuration for production in order to reduce variability in performance.

If you are planning on running this on dedicated hardware in production, then the best option is to, again, run the performance tests on as close to a similar setup as you can. If you are going to VMs for testing because of resource constraints, then I would suggest setting apportioning resources to your VM so that they are fixed as they would be on the dedicated hardware.