How can QA staff test caching logic that they can't see?

Question

I just implemented a caching layer in my web application, and now I'm wondering how QA is supposed to test it, since caching is transparent to the user.

One idea I have is to put logging in the methods that invoke the code that populate the cache, and record when an object is pulled from cache and when it requires recreation from the database, and then the testers could view the logs to see if, for example, a certain object is reloaded from the db every 10 minutes, instead of every page view.

But can anyone suggest some better practices for this situation?

Answer 1

One question is whether the cache itself is really a requirement that should be tested by QA. Caching improves performance, so they could test the difference in performance to ensure it meets some requirement.

But good idea to have some testing around caching, whoever is responsible for it. We used performance counters. If your cache system takes advantage of these, they are straightforward. If there is any way to get a count from the cache itself, that is another option.

Using your approach is nice too. If any of these are wrapped in automated tests that check the results, then no one has to look through logs to find answers.

Answer 2

You implemented a cache (I assume) because the system wasn't performing well enough. That is something that is relevant for the user. Here are things that QA can check:

• That the system, after the cache was introduced, is still correct. This means they should attempt to trick the cache into providing stale data, which is something QA can verify, and make sure nothing else was broken.
• That the system now meets performance threshholds that it was not meeting when you decided to improve performance. They can compare the old measurements and compare them to the new ones.

Remember, users (and, by extension, QA) don't care about how you solved a problem. They only care that the problem was solved and didn't create new problems. This is true whether you implemented caching, improved String parsing, did a hardware upgrade, or sprinkled magic fairy dust on the server.

Answer 3

Burying important business logic or system state deep in a black box makes verifying correct system behavior difficult. It's easier to exhaustively test the behavior of a single component in the system than the entire system. I favor exposing such things explicitly through some mechanism so it can be unit / regression / integration / QA tested in some meaningful fashion.

One option with a cache would be to expose a special page that gives some details on the cache (contents, state, etc). This can assist with debugging in development and potentially in production. QA can also use this page to create test cases for the cache if they are given details on what the expected behavior of the cache is. Using performance counters and/or log files to explicitly document the cache behavior is another less visible but viable approach.

Note that this approach is not a substitute for end-to-end performance testing. This is a mechanism to ensure the cache itself behaves correctly. Performance testing should be used to determine if caching has the intended aeffect on performance.

Also note that swapping out components of the system with new ones implementing the same interface like introducing a cache can be a destabilizing and deceptively complex change. With the cache example, you are introducing state in what was previously stateless, which can create bugs that are harder to find or reproduce. Such a change should always be accompanied with full regression testing to verify expected system behavior.

Answer 4

Test performance, as indicated in Andy's answer.

I have found that the biggest obstacle to implementing caching (and performance) at many organizations is actually having an environment where you can do good performance testing and run tests for various real world load and performance tests.

To have this you should set up a performance testing environment that, as closely as possible, and allowing for costs, mirrors production. This will probably NOT be your current development environment which should be be smaller and more self-contained to allow for rapid application development. Development environments also tend to use caching less and so don't represent production well for performance testing.

In the performance testing environment the app should be running in production 'mode'. You should more than one server if production does, database connection pool and caching should be set for a production environment, etc.

You'll also want to consider a tool to help with load testing.
jmeter is very popular though I find it quite unfriendly and primitive to use.
Another route I've used is to just url curl's with a ruby script.

To be clear

• base line performance testing is for testing the time ONE request makes.
• load testing is similar to performance testing but looks at response when the system is also under load from other requests.

You may also find the following links helpful:

• Best practices for unit testing methods that use cache heavily?
• Load Test Methodology for a cache
• Is there any way to test how will the site perform under load
• http://www.procata.com/cachetest/

Answer 5

Remember to get the testers to reboot the servers and check that the data they have entered is still there. I saw a system that had many man months of testing, fail when that was done!

The hardest part to test is that however the data is updated, there is never any out-of-date results returned from the cache.

This can be partly helped by always using the data in the cache to populate the confirmation page a user sees after they have made a change. E.g don’t use the object you used to update the database, but request the data from the cache, that then request it from the database. A bit slower, but much more likely to show up bugs quicker.

Answer 6

This one actually has a very straightforward answer and it is related to the level of caching. What you will observe when the caching is correct is the absence of requests at the target of the requests. So, it comes down to the hackneyed QA phrase of "expected results."

If implementing a cache at the web tier, then I would expect that items subject to the caching would only show up once for each tested user session (if implementing client cache) or once for multiple users (if implementing a CDN style cache). If you are implementing a cache at the data tier for common results, then I would expect to see a high cache hit ratio in your caching tier along with an absence of queries in the profile log for the data tier.

etc...

Answer 7

Some things are better tested by a programmer, perhaps the one who wrote the code, using unit tests. Testing the correctness of your caching code is one of those things. (From the way you ask this question, I presume that your QA people treat the application as a "black box" and test it through its external interface.)

Answer 8

Caching logic is something that should be unit tested by the developer as QA mainly does black-box testing.

QA would only care about the performance aspects or whatever fix you implemented thus, you can provide QA with a mechanism to enable/disable caching or whatever mechanism you used to improve performance, and then they can verify the performance difference. Of course QA could also just verify an old release against your performance-improved one as well.

Answer 9

When I was testing caching solution we have implemented then what we have tested basically the performance. We have done that caching solution for XML results and after the cache it takes very low time to give the response. Also we have checked it with the server log by checking the log entries.