I'm currently writing a load test for a cache which should test how a cache will react to persistent requests. A colleague and I had differing opinions on how this load testing should be performed.
I believe that a load test should be as random as possible. It should model real-world load as much as possible, and the way towards that is randomality. So I have created this random test as follows:
- Test data is held in spreadsheets and is loaded into TestRunner objects at startup
- The test data is not random
- The load test will run 10 random TestRunners in individual Threads at the same time
- The object returned by the cache will be tested to make sure it is sensible, it is not thoroughly tested
- Any tests that fail will be output at the end and each test has a unique ID to easily find failed tests
- At random intervals, the cache will be cleared to model the real-world requirement of the cache being cleared at any time.
- The load test will run for a configurable amount of time
My colleague's idea of what a load test should do is:
- Test data is held in spreadsheets and is loaded into TestRunner objects at startup
- All TestRunner objects are run in a sequential manner
- Each time the load test is run, it will run the TestRunner objects in the same order
Which methodology do you feel would produce the most reliable load test?
I personally think the random test will produce a more reliable load test as it will model real-world usage. It is not known what order requests will come in when this is released to production, so it should be tested with that unknown element taken into account. However, running all tests in the same sequence each time will make any failures reproducable, which is important during testing.