I hope you bear with me here. It's difficult to explain this problem, especially so without direct reference to the specific domain.

I work on a component of a larger system which performs computationally expensive pre-processing in order that the "real-time" component can perform it's functions quickly. My product effectively creates a big lookup dataset for the main component to query.

My product computes the results of a finite set of "scenarios" (a few hundred thousand) any one of which may be performed by a customer. The component uses a k-shortest paths algorithm and a number of rules engines to consider each scenario and build up the lookups, operating on reference data that represents the problem domain.

For the scenarios that are very commonly performed by our customers the test cases are easy; the expected outcomes are easy to devise and assert acceptance tests. However, for the scenarios that are not "popular", the expected outcomes are difficult to establish and in any case their outcomes may be changed by changes to the reference data which are virtually impossible to predict. In other words, changes to the reference data changes the inputs to the shortest-path algorithm which in turn may change the output.

Creating acceptance tests for these unpopular cases is also tricky, because the business "doesn't know what it doesn't know", and so has no baseline against which it can be tested.

Whenever a tester comes back to me with a defect, 99.9% certainty it's because conditions in the reference data have changed, rather than because of any code changes. I therefore spend a load of time chasing defects that are "data issues". I think what confuses the situation is that the reference data is key to the functionality of the system, and the system without a specific set of reference data would produce no results that had any meaning to the business. I could create a set of "fake" reference data against which to test, but any assertion against this engineered reference data would be invalid in the real world.

I would be very interested to hear any perspectives on this especially from those who have encountered similar problems. I am anxious not to simply shrug my shoulders and reply "data issue" but get to the root of the problem and work out how to build and test this thing in a reliable, repeatable and valuable way.

  • I'm a little unclear on what you mean by "data issues." Who fixes the data issues, you or the customer? Do these data issues represent new test cases, and could you simply incorporate those test cases as you receive them from the customer? Dec 9, 2013 at 19:07
  • @Robert Harvey - I guess the data is the responsibility of the business to maintain, but it is very finely balanced. If the weight of one edge on the graph (which is setup by reference data) is changed then it could have implications in many other scenarios which cannot be foreseen. Seems as though the design itself isn't allowing the application to be robust enough.
    – Matt
    Dec 9, 2013 at 19:19
  • If the reference data changes sufficiently to change the output, how does the tester know it's a defect as opposed to simply the correct result for this new input?
    – Telastyn
    Dec 9, 2013 at 20:13
  • @Telastyn Well, this is the problem - the testers don't know this. All they know is that the system once did X, but now does Y (or doesn't do X anymore), which to their eyes is either a failure or a regression.
    – Matt
    Dec 10, 2013 at 8:37

3 Answers 3


A good approach to this (assuming your company has the stomach for it) would be to model several major classes of customer data scenarios as testable requirements, write testable requirements for those, and then write tests for those requirements. To identify the generalizations that define the most common of the uncommon scenarios, in other words, and make sure your software adequately addresses them.

This would have several benefits. Your company would understand their software better, have the tools for creating better customer documentation and service, and maybe even identify ways to utilize the software that were previously unforeseen.

Any such effort would have to be driven directly by customer feedback, so that you can justify the effort for those scenarios that have a genuine customer interest. In other words, you don't just go looking for scenarios that might be useful.


I think you discount fake data to quickly. The point of testing is to show that your k shortest path implementation is in fact finding shortest paths, not that it works on the exact user input. If it works on good enough fake data it will work on real data modulo any errors in the data or your understanding of the data.

It sounds like you need two rounds of testing, one for your algorithm (with fake data meant to exercise every code path not just the ones the customer happens to use) and one for the data to verify its integrity and compatibility(that should be treated as data errors until shown otherwise).

  • Is it practical to meaningfully exercise every code path in a "shortest paths" algorithm, under every reasonable data scenario? How many permutations are possible? The customer may not accept "my algorithm works, you just have bad data" as a satisfactory explanation. Dec 9, 2013 at 19:19
  • @Robert Harvey - Exactly. The business have a single model and state their requirements in terms of nodes on that model, not in terms of generalisations. There are about 410,000 different scenarios possible; only about 70,000 of which are actually popular. Although I could in all good conscience reject "data issues" I want to find a better way of testing this system. Maybe there isn't one...!
    – Matt
    Dec 9, 2013 at 19:27
  • @Matt: Thanks for the additional information. That's enough for me to write an answer. Dec 9, 2013 at 19:29
  • 1
    @RobertHarvey It depends on their system entirely. It shouldn't be necessary to create all possible graphs to show that a graph walking algorithm is working correctly. You should just need a few graphs with all relevant to the code variations.
    – stonemetal
    Dec 9, 2013 at 19:33
  • 2
    Consider having the algorithm explain how it arrives to a particular result (on top of providing the result)? A customer might be ok with 'I got <foo> because <bar>', when they're not with 'magic happen, and the answer is <foo>'.
    – ptyx
    Dec 9, 2013 at 21:58

In this situation the data is really part of the algorithm. Just as changes to the code need to be version controlled and unit tested, and the data that drives it needs to go through version control, some sort of verification, and then the two parts should be integration tested.

Sounds like the people who change the reference data need to be defining what their changes are intended to achieve (acceptance criteria), and then testing for those criteria.

You should also be a part of that process, and part of the solution. How can you help them be more successful? Since you don't say anything about the people who change the reference data there is not much to go on, but I'm guessing they (and you) don't see themselves as IT people. So you'll have to help them to see the big picture, where their data and your algorithm work together to produce their required results.

PS I would also add that the code and the reference data should be regression tested separately.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.