33

I just implemented a caching layer in my web application, and now I'm wondering how QA is supposed to test it, since caching is transparent to the user.

One idea I have is to put logging in the methods that invoke the code that populate the cache, and record when an object is pulled from cache and when it requires recreation from the database, and then the testers could view the logs to see if, for example, a certain object is reloaded from the db every 10 minutes, instead of every page view.

But can anyone suggest some better practices for this situation?

37

One question is whether the cache itself is really a requirement that should be tested by QA. Caching improves performance, so they could test the difference in performance to ensure it meets some requirement.

But good idea to have some testing around caching, whoever is responsible for it. We used performance counters. If your cache system takes advantage of these, they are straightforward. If there is any way to get a count from the cache itself, that is another option.

Using your approach is nice too. If any of these are wrapped in automated tests that check the results, then no one has to look through logs to find answers.

  • +1 for testing performance instead of caching. What business value is caching on its own? (None.) Surely you are working towards some noticeable impact on something - why else would you spend the time implementing it? Test for that impact. – alexanderbird Apr 25 '16 at 16:55
74

You implemented a cache (I assume) because the system wasn't performing well enough. That is something that is relevant for the user. Here are things that QA can check:

  • That the system, after the cache was introduced, is still correct. This means they should attempt to trick the cache into providing stale data, which is something QA can verify, and make sure nothing else was broken.
  • That the system now meets performance threshholds that it was not meeting when you decided to improve performance. They can compare the old measurements and compare them to the new ones.

Remember, users (and, by extension, QA) don't care about how you solved a problem. They only care that the problem was solved and didn't create new problems. This is true whether you implemented caching, improved String parsing, did a hardware upgrade, or sprinkled magic fairy dust on the server.

  • 1
    Stating that QA doesn't care how you solve the problem is a very simplistic view of what QA is interested in. They care about whether you actually improved performance, introduced additional technical debt, risk, or defects, etc. – Eric Mar 18 '15 at 21:14
  • 4
    @Eric In software development, "QA" as a group does not generally refer to developers/engineers. Technical debt is a developer/engineer concern (and a management concern since it can impact timelines, costs, etc.). The same is true of risk. QA's job is to make sure that the software meets the requirements (and possibly incidentally help in the process of flushing out requirements that were unclear). If they care about the implementation at all, it should only be as a means of figuring out creative ways to break the system. – jpmc26 Mar 18 '15 at 21:41
  • 3
    @Eric I am confusing nothing. "QA" does refer to testers in software. You are interpreting the term so broadly that it should include the entire company developing the piece of software and even the client if it's a system built custom for them, since everyone has a hand in quality. – jpmc26 Mar 18 '15 at 22:34
  • 1
    @Eric Please don't make me argue how language and semantics work with you. I challenge you to find 5 instances on this StackExchange where the term is used that way in less than 30 minutes. Besides, even by that definition, the best a separate QA group can do to address problems beyond the product itself is to impose policies on developers, and when policy and process are elevated above making good products, you usually end up with bad products. Also, I can assure you that the "QA" people I work with most definitely are testers, and have little say about my development process. – jpmc26 Mar 18 '15 at 22:44
  • 4
    @Eric: there is nothing stopping a company or group of people defining "QA" to be about the more holistic view. However, in my experience (across 5 very different companies) the QA stage as named - and those who specialise in it - in software development is usually focussed on black-box testing of system behaviour. Whilst we also may have "Quality Control", "Kwalitee" and more invented names in order to cover specialist angles on wider quality issues. – Neil Slater Mar 19 '15 at 8:12
3

Burying important business logic or system state deep in a black box makes verifying correct system behavior difficult. It's easier to exhaustively test the behavior of a single component in the system than the entire system. I favor exposing such things explicitly through some mechanism so it can be unit / regression / integration / QA tested in some meaningful fashion.

One option with a cache would be to expose a special page that gives some details on the cache (contents, state, etc). This can assist with debugging in development and potentially in production. QA can also use this page to create test cases for the cache if they are given details on what the expected behavior of the cache is. Using performance counters and/or log files to explicitly document the cache behavior is another less visible but viable approach.

Note that this approach is not a substitute for end-to-end performance testing. This is a mechanism to ensure the cache itself behaves correctly. Performance testing should be used to determine if caching has the intended aeffect on performance.

Also note that swapping out components of the system with new ones implementing the same interface like introducing a cache can be a destabilizing and deceptively complex change. With the cache example, you are introducing state in what was previously stateless, which can create bugs that are harder to find or reproduce. Such a change should always be accompanied with full regression testing to verify expected system behavior.

3

Test performance, as indicated in Andy's answer.

I have found that the biggest obstacle to implementing caching (and performance) at many organizations is actually having an environment where you can do good performance testing and run tests for various real world load and performance tests.

To have this you should set up a performance testing environment that, as closely as possible, and allowing for costs, mirrors production. This will probably NOT be your current development environment which should be be smaller and more self-contained to allow for rapid application development. Development environments also tend to use caching less and so don't represent production well for performance testing.

In the performance testing environment the app should be running in production 'mode'. You should more than one server if production does, database connection pool and caching should be set for a production environment, etc.

You'll also want to consider a tool to help with load testing.
jmeter is very popular though I find it quite unfriendly and primitive to use.
Another route I've used is to just url curl's with a ruby script.

To be clear

  • base line performance testing is for testing the time ONE request makes.
  • load testing is similar to performance testing but looks at response when the system is also under load from other requests.

You may also find the following links helpful:

2

Remember to get the testers to reboot the servers and check that the data they have entered is still there. I saw a system that had many man months of testing, fail when that was done!

The hardest part to test is that however the data is updated, there is never any out-of-date results returned from the cache.

This can be partly helped by always using the data in the cache to populate the confirmation page a user sees after they have made a change. E.g don’t use the object you used to update the database, but request the data from the cache, that then request it from the database. A bit slower, but much more likely to show up bugs quicker.

1

This one actually has a very straightforward answer and it is related to the level of caching. What you will observe when the caching is correct is the absence of requests at the target of the requests. So, it comes down to the hackneyed QA phrase of "expected results."

If implementing a cache at the web tier, then I would expect that items subject to the caching would only show up once for each tested user session (if implementing client cache) or once for multiple users (if implementing a CDN style cache). If you are implementing a cache at the data tier for common results, then I would expect to see a high cache hit ratio in your caching tier along with an absence of queries in the profile log for the data tier.

etc...

0

Some things are better tested by a programmer, perhaps the one who wrote the code, using unit tests. Testing the correctness of your caching code is one of those things. (From the way you ask this question, I presume that your QA people treat the application as a "black box" and test it through its external interface.)

0

Caching logic is something that should be unit tested by the developer as QA mainly does black-box testing.

QA would only care about the performance aspects or whatever fix you implemented thus, you can provide QA with a mechanism to enable/disable caching or whatever mechanism you used to improve performance, and then they can verify the performance difference. Of course QA could also just verify an old release against your performance-improved one as well.

-4

When I was testing caching solution we have implemented then what we have tested basically the performance. We have done that caching solution for XML results and after the cache it takes very low time to give the response. Also we have checked it with the server log by checking the log entries.

  • 3
    I am not quite sure I understand what you are saying, or what your answer says that is not in the seven previous answers. – user22815 Nov 19 '15 at 13:58

protected by user22815 Nov 19 '15 at 13:58

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.