Load Test Methodology for a cache

Question

I'm currently writing a load test for a cache which should test how a cache will react to persistent requests. A colleague and I had differing opinions on how this load testing should be performed.

I believe that a load test should be as random as possible. It should model real-world load as much as possible, and the way towards that is randomality. So I have created this random test as follows:

• Test data is held in spreadsheets and is loaded into TestRunner objects at startup
  • The test data is not random
• The load test will run 10 random TestRunners in individual Threads at the same time
• The object returned by the cache will be tested to make sure it is sensible, it is not thoroughly tested
  • Any tests that fail will be output at the end and each test has a unique ID to easily find failed tests
• At random intervals, the cache will be cleared to model the real-world requirement of the cache being cleared at any time.
• The load test will run for a configurable amount of time

My colleague's idea of what a load test should do is:

• Test data is held in spreadsheets and is loaded into TestRunner objects at startup
• All TestRunner objects are run in a sequential manner
• Each time the load test is run, it will run the TestRunner objects in the same order

Which methodology do you feel would produce the most reliable load test?

I personally think the random test will produce a more reliable load test as it will model real-world usage. It is not known what order requests will come in when this is released to production, so it should be tested with that unknown element taken into account. However, running all tests in the same sequence each time will make any failures reproducable, which is important during testing.

Answer 1

Do you have a way to reset the data once the test is run (if this is even necessary)?

If so, what about running the non-random test first - to look for failures when run that way (and possible anomalies caused by the data itself)? Then, after resetting (if necessary), run the random tests to reflect the real world load.

Answer 2

Multi-thread tests are a pain. It might simulate a real-world scenario but will only tell you that 'There is a problem'. You will not be able to reproduce the test run and, most of the times, this means that you will not be able to find the bug.

The random execution model will reveal strange bugs that don't arise on normal usage - otherwise you would know about them just by 'playing' with the product. That's why the chance of reproducing the same bug with another random run is very small (unless you have infinite time :) ).