Writing tests for API that results depends on table with millions of rows

Question

I think my problem is very common but I couldn't phrase it correctly in the title.

I have a Django API that returns some information about companies (like address, partners, situation, etc) given its unique identifier. I would like to run tests on it because it is very unstable given that it depends on a lot of data that I sometimes I can't predict (like null addresses, invalid situations, no partners, etc) and are hard to predict while I'm coding.

I don't know how to run efficient tests on it given that it is not feasible to run it for every company (there are millions of it). I think maybe an idea is to do it by sampling, but this would lead to unpredictable tests since it might work for some companies and others not, running tests for only a subset of predefined ones wouldn't be ideal because it might lead to biased tests (they pass on those but there are some companies that it doesn't).

Does anyone have an idea for a solution?

Answer 1

When manufacturing test data for unit tests you want to ensure that the edge cases (sometimes called corner cases) are correct. An edge case includes data that should generate an error. You'll want to include data that hits each edge case you know about.

You do want to ensure that the test can properly differentiate between a passing case and an non-passing case. You can either include the expected result as one of the data elements you pass in to your parameterized test, or you can have separate tests for good data and bad data. Both approaches work well, and the choice you take depends on the similarity of the structure of the tests.

In general you want your data to have these attributes:

• The data should be representative. In other words, you don't need more than one record for each type of failure.
• The data should be realistic. I.e. it should look like valid data, but not actually use real data. That's more for a legal protection, particularly if it's data that can be considered private like people's addresses.
• The data should be minimal. In short, you want just enough cases to catch the problems you experience.

After you've exercised those cases in the subset of curated data you have for your unit test, you should have a high level of confidence that the code will work. However, if you run into a real record where you code doesn't behave as expected, you'll want to add that into the test data. Make sure you understand what the problem is. For example, you might not be handling special characters like embedded quotes or international characters properly. Those are additional edge cases you need to test for.

Answer 2

For getting an API stable, black-box testing with real data is only one instrument.

At least equally important is learning how to write robust code (code which expects systematically "bad" data, missing data, null values, non-ASCII characters etc), code reviews/audits, and designing specific tests immediately whilst you code, using the white-box knowledge of what can go wrong internally, and/or using test-driven development. Also, systematic internal consistency checks and logging can help. Whenever code processes external data, avoid any assumptions that the data is in "good shape" - better assume it could be broken somehow.

That it usually far more effective than writing code for the "happy path" first and then throwing a million arbitrary records at it afterwards to find out which edge cases have been forgotten to deal with. If you put more focus on the former instruments, taking a certain percentage of samples from production data will probably be enough. It is also a good idea if you find an error by using these samples, not just to fix it and make a new test case for it, but also check if the error indicates some forgotten robustness tests in other areas of the code.

In the end, be aware no test can proof the absence of bugs, only their presence. Even if you could make a test run for your API against a million of records today, you don't know if the next record entered tomorrow will contain an unexpected case. But if you know your code checks for unexpected things on multiple levels, you can gain confidence your code will be able to handle that case without crashing the system.