7

I'm implementing a metadata parser of image files from all formats. I want to write tests for it. One trivial way to do so is to have test image files of all formats as a resources for the tests, and actually to read them as input. This approach may work but as far as I understand from unit test methodology, unit-tests shouldn't perform I/O. Is it a good way to do so or are they any alternatives?

  • 2
    Let the parser work on a block of memory that contains the image data. – πάντα ῥεῖ Mar 18 '18 at 8:29
  • @πάνταῥεῖ Yes, this is the second option. The problem with it is that some formats (e.g. RAW ) may contain up to 50MB of data. Is it feasible to generate such a block for test? – Sanich Mar 18 '18 at 8:33
  • 1
    This question once troubled my mind too. – ferit Mar 18 '18 at 10:02
  • 4
    Probably, TDD's worshipers will curse me after say this but, not every single line of code worth a UT. And much less a dogmatic approach of TDD. If your solution is light and simple, the complexity of the tests should not exceed that complexity. Otherwise, the costs of the QA exceeds the benefits. In other words, if programming and running the tests takes longer than coding and evolving the solution, then you are wasting time and money. That said, I would weight which of the "moving parts" worth to be tested unitarily and make everything else e2e test. Just make them deterministic. – Laiv Mar 18 '18 at 14:53
  • 1
    You can split the tests into small format parsing tests and larger load testing tests. You don't need a 50MB file to ensure that the header has the information you expect. However, in your integration tests, you can use 50MB+ files in your tests where I/O is allowed--just to make sure things still work. – Berin Loritsch Mar 19 '18 at 16:23
9

I want to write tests for it.

What you are intending to test?

I want to use TDD. I'm refactoring a parser and want to test the 'parse()' method.

So the aim is to clean things up.

I would argue that refactoring legacy code isn't 100% compliant w/ TDD.

Bad code restricts testing.

More importantly - intention to clean it up (the drive - reason for changing code) differs from original intention for code to do whatever business domain stuff.


step 1

I would start with a sloppy integration test/s that covers most of the functionality.

Feed tests crude input - e.g. those 50mb resource files.
Ask only polished output and ignore internal stuff.

It's actually important - higher test abstractedness is what loosens implementation restrictions.

That will give you a safety net so you can open up code for refactoring w/o fear.

step 2

Once you have that - you are ready to actually go in & refactor.

Read the code. Start small. (good book)

Things like code formatting, removal of excess white-space, removal of too verbose variable prefixes.

Then move forward to structural changes - extract methods, interfaces, classes where needed.
And don't just divide & conquer - try combining stuff where it "makes sense" ™.

Only with decent code structure you will be able to write unit tests for isolated units of functionality.

If the integration test you started with performs well enough - I wouldn't even bother trying to build unit test network.

Either way - proper code structure will lead you to natural & easy to stub I/O seam.

Once the network of unit tests is strong enough - remove integration test/s.
Or stub the input the same way as in unit tests (sort of devalues integration test).

4

as far as I understand from unit test methodology, tests shouldn't perform I/O

You have to distinguish which kind of tests you are talking about:

  • if you want to use TDD with small "write test" - "write code" - "refactor" cycles, then you need very quick unit tests, ideally with small data sets and no I/O.

  • when you want to do acceptance or integration testing, especially by using several external images, then using files & I/O is not just perfectly fine, but probably required.

So decide on the basis which kind of tests you are writing.

  • I want to use TDD. I'm refactoring a parser and want to test the 'parse()' method. This is a unit-test. (By no means an acceptance or integration). So the quick way to write a test is to read the input from a test file. But as might be implied from your answer to my other question, the better but much longer way is to use a cache-free solution. – Sanich Mar 18 '18 at 9:44
  • 4
    "Parse a complicated file format" doesn't sound like a unit test to me - you'd have separate independently testable components for the header, the body and whatever else. You unit test those components, mock them out when unit testing the whole parser, and then integration test the whole parser. – Philip Kendall Mar 18 '18 at 10:46
  • "the better but much longer way is to use a cache-free solution" - no, you got me wrong on this: I was assuming if a cache-free solution is less (or at least not more) effort than a solution with a cache. Let me reword my other answer to make this more clear. – Doc Brown Mar 18 '18 at 13:41
4

This approach may work but as far as I understand from unit test methodology, unit-tests shouldn't perform I/O

I think you are overlooking the elephant in the room here: your parser should not perform I/O.

Your parsers job is parsing data, not disk I/O. Whatever language you are using, it probably has the concept of a stream. That is where your parser gets data from. And whether that happens to be a FileStream or MemoryStream or even std::cin is not your parsers business.

So back to your tests: It does not matter. Your tests will have the data to test your parser. And whether that is written on disk in separate files or resource manifest streams or hardcoded byte-arrays... does not really matter. You CI automation should be able to work with it and that's all you need to know. In the end, the data is on disk. Where else would it be? Even your executable code is loaded from disk.

So to sum it up: make your parser independent from disk access. Separation of concerns. Then have your tests load data in the way that is most elegant for your CI solution so your tests don't fail when someone else runs them on their machine (someone else might as well be your CI agent). How you achieve that is a detail that nobody really cares about as long as it works.

  • 1
    Exactly, I may have my test pull something from a file when it is convenient (big chunk of data, a large list, etc.), but not be testing I/O functionality. – JeffO Mar 20 '18 at 16:07
1

When testing parsers, there's at least two modes of testing that need to happen:

  • Testing in the micro: the parser knows how to parse a phrase (a set of bytes or characters)
  • Testing in the macro: the parser needs to parse a whole set of phrases

If your parser is modal, then you'll have tests to ensure the mode transitions are handled properly.

The bottom line is that they are two different sets of tests. Usually I end up splitting the work this way:

  • Unit Tests are for testing individual phrases
  • Integration Tests are for testing whole files, and problem cases

Usually I can set up the bytes in memory for the micro tests, and use the filesystem for the integration tests. This has served me well. That said, creating test phrases for textual parsers (the bulk of what I've done) is much easier than doing the same for binary parsers. I have done some binary parsing, so my test case will likely have what looks like magic arrays of bytes, but it works.

1

If I understood you correctly, you have two goals

  • add new features/functionality/refactoring to the existing code
  • While extending/refractoring the existing code, make sure that the current functionality does not break (aka "regression test" or saftey net)

If I were you, I would start with the "make sure that the current functionality does not break" part as a file based integrationtest with

  • a folder of example images
  • for each image there is a textual representation of the exprected result.

This (probably slow) regression test would iterate over all image files and compares parsing with the expected file.

Once this is working you can start thinking about TDD with new features, which is something completly different from the proposed regression test.

For the TDD part of the question we cannot help you without seeing code. (question is too broad)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.