Burying important business logic or system state deep in a black box makes verifying correct system behavior difficult. It's easier to exhaustively test the behavior of a single component in the system than the entire system. I favor exposing such things explicitly through some mechanism so it can be unit / regression / integration / QA tested in some meaningful fashion.
One option with a cache would be to expose a special page that gives some details on the cache (contents, state, etc). This can assist with debugging in development and potentially in production. QA can also use this page to create test cases for the cache if they are given details on what the expected behavior of the cache is. Using performance counters and/or log files to explicitly document the cache behavior is another less visible but viable approach.
Note that this approach is not a substitute for end-to-end performance testing. This is a mechanism to ensure the cache itself behaves correctly. Performance testing should be used to determine if caching has the intended aeffect on performance.
Also note that swapping out components of the system with new ones implementing the same interface like introducing a cache can be a destabilizing and deceptively complex change. With the cache example, you are introducing state in what was previously stateless, which can create bugs that are harder to find or reproduce. Such a change should always be accompanied with full regression testing to verify expected system behavior.