I'm implementing a metadata parser of image files from all formats. I want to write tests for it. One trivial way to do so is to have test image files of all formats as a resources for the tests, and actually to read them as input. This approach may work but as far as I understand from unit test methodology, unit-tests shouldn't perform I/O. Is it a good way to do so or are they any alternatives?
I want to write tests for it.
What you are intending to test?
I want to use TDD. I'm refactoring a parser and want to test the 'parse()' method.
So the aim is to clean things up.
I would argue that refactoring legacy code isn't 100% compliant w/ TDD.
Bad code restricts testing.
More importantly - intention to clean it up (the drive - reason for changing code) differs from original intention for code to do whatever business domain stuff.
I would start with a sloppy integration test/s that covers most of the functionality.
Feed tests crude input - e.g. those 50mb resource files.
Ask only polished output and ignore internal stuff.
It's actually important - higher test abstractedness is what loosens implementation restrictions.
That will give you a safety net so you can open up code for refactoring w/o fear.
Once you have that - you are ready to actually go in & refactor.
Read the code. Start small. (good book)
Things like code formatting, removal of excess white-space, removal of too verbose variable prefixes.
Then move forward to structural changes - extract methods, interfaces, classes where needed.
And don't just divide & conquer - try combining stuff where it "makes sense" ™.
Only with decent code structure you will be able to write unit tests for isolated units of functionality.
If the integration test you started with performs well enough - I wouldn't even bother trying to build unit test network.
Either way - proper code structure will lead you to natural & easy to stub I/O seam.
Once the network of unit tests is strong enough - remove integration test/s.
Or stub the input the same way as in unit tests (sort of devalues integration test).
This approach may work but as far as I understand from unit test methodology, unit-tests shouldn't perform I/O
I think you are overlooking the elephant in the room here: your parser should not perform I/O.
Your parsers job is parsing data, not disk I/O. Whatever language you are using, it probably has the concept of a stream. That is where your parser gets data from. And whether that happens to be a
MemoryStream or even
std::cin is not your parsers business.
So back to your tests: It does not matter. Your tests will have the data to test your parser. And whether that is written on disk in separate files or resource manifest streams or hardcoded byte-arrays... does not really matter. You CI automation should be able to work with it and that's all you need to know. In the end, the data is on disk. Where else would it be? Even your executable code is loaded from disk.
So to sum it up: make your parser independent from disk access. Separation of concerns. Then have your tests load data in the way that is most elegant for your CI solution so your tests don't fail when someone else runs them on their machine (someone else might as well be your CI agent). How you achieve that is a detail that nobody really cares about as long as it works.
as far as I understand from unit test methodology, tests shouldn't perform I/O
You have to distinguish which kind of tests you are talking about:
if you want to use TDD with small "write test" - "write code" - "refactor" cycles, then you need very quick unit tests, ideally with small data sets and no I/O.
when you want to do acceptance or integration testing, especially by using several external images, then using files & I/O is not just perfectly fine, but probably required.
So decide on the basis which kind of tests you are writing.
When testing parsers, there's at least two modes of testing that need to happen:
- Testing in the micro: the parser knows how to parse a phrase (a set of bytes or characters)
- Testing in the macro: the parser needs to parse a whole set of phrases
If your parser is modal, then you'll have tests to ensure the mode transitions are handled properly.
The bottom line is that they are two different sets of tests. Usually I end up splitting the work this way:
- Unit Tests are for testing individual phrases
- Integration Tests are for testing whole files, and problem cases
Usually I can set up the bytes in memory for the micro tests, and use the filesystem for the integration tests. This has served me well. That said, creating test phrases for textual parsers (the bulk of what I've done) is much easier than doing the same for binary parsers. I have done some binary parsing, so my test case will likely have what looks like magic arrays of bytes, but it works.
If I understood you correctly, you have two goals
- add new features/functionality/refactoring to the existing code
- While extending/refractoring the existing code, make sure that the current functionality does not break (aka "regression test" or saftey net)
If I were you, I would start with the "make sure that the current functionality does not break" part as a file based integrationtest with
- a folder of example images
- for each image there is a textual representation of the exprected result.
This (probably slow) regression test would iterate over all image files and compares parsing with the expected file.
Once this is working you can start thinking about TDD with new features, which is something completly different from the proposed regression test.
For the TDD part of the question we cannot help you without seeing code. (question is too broad)
A parser is a syntax-directed translator that converts a set of input words conforming to a base grammar to output "words", the words normally taking the form of a sequence of actions.
Reverse the parser. Instead of it recognizing a base language and producing outputs for the input words it recognizes; have it generate the words of the base language and feed that into the parser. That means that at each state of the original parser, instead of accepting an input word and testing its validity and carrying appropriate actions for the input accepted, the reverse-parser generates the input word (both valid and invalid) using a probability metric. So it becomes a stochastic automaton feeding off of a random number generator that generates the base language + errors (for stress testing).
Thus, the tester is just the parser itself - run in reverse - to generate words in the base language. In effect, it becomes a translator that converts the random stream of a random number generator into the output language that, itself, is (apart from intentionally generated errors) the input language of the parser.
The test suite then consists of a set of automatically generated random number seeds. Each update of the parser is regression tested against its predecessor on the same set of automatically generated test cases.
The test cases are generated with the reversal of the original parser. But after sufficient development, you may then reverse the upgraded parser and use that as the test generator instead. Then, the test-generator has to be regression-tested by running it against the previous test-generator to ensure it produces the same test cases off the random number generator. In the process, you may design the upgraded parser so that it can be more easily reversed.
Thus, both tester and parser are bumped up side by side.