I have a general question about testing search applications, and what I'm looking for is pointers to resources on the topic that I can go and research on my own. I've tried semi-informed, semi-undirected googling, but that's yielding a lot of distractions and blind alleys (or maybe my search skills aren't all that sharp).
A bit of setup first. When I say "a search application", what I mean this:
- you have some data sources which you can put together in a search index
- your application has an API that takes as input a search query (keywords and optionally other stuff), and its output is a relevance-ranked list of results from the search index.
- there is a whole bunch of business logic on top of just retrieving the results from the index - the final result set in the output could have a large edit distance from the original result set from search.
- assume that in real life, the index is large and takes a while to build
The task is to write tests for the application. The basic structure of a test is "given search request X, I expect response Y consisting of relevance-ordered results". The problem, therefore, is: what is a good strategy for generating the underlying data for the tests?
Here are some approaches I'm aware of (and have used in practice):
- Don't generate test data. Start with a real index, and apply targeted modifications to it to "introduce" edge cases for your tests as needed. Upside: close to real life. Downsides: large test index; has to be rebuilt every time some change is made to the indexing scheme; most of it is unused by existing test cases.
- Generate fake data such that for each request X there is a well-defined, intentionally constructed set of results Y that will be returned. Upsides: full control over the search data; only as much data as needed for the tests, quicker and easier to change. Downsides: still have to rebuild everything in the indexing scheme changes; not necessarily realistic data, which may leave aspects of the system untested or under-tested; too much flexibility and test-specific domain knowledge separate from the real-life domain of the application.
Actually that's where my current knowledge ends. Something tells me there is either a nice middle ground which allows for testing flexibility without deviating from how the application works in real life, or a completely different testing approach that does away with these concerns. What approaches might you consider?