I wrote a small INI file parser as a library which I want to use in a bigger project. Following good practice I decided I should write test cases, too. But I fail to find a good start.

  • The library is extremely small (1 source file for the implementation, 1 header)
  • The library has few public methods. (parse(), get_sections(), get_value(section, key))
  • Most logic is private to the implementation, therefore not trivially accessible for tests.
  • The input for a test would be an INI file and I don't want to write multiple test INI files.

So the first test I'd write is:

  • Call the parse method on a provided test input INI file and check that it was successful.
  • Hardcoding expected sections and key-value pairs in the test itself and check that all of them are there.
  • Also check that no other sections and key-value pairs were extracted.

I am unhappy with that. Maintaining the expected output and the input files is quite a burden. The tests will fail when you update the input but not the expectations which I think is a bad design decision for tests.

There are many guidelines for writing good tests out there, but I always find it hard to apply them. I guess experience is the key. So maybe you can guide me to some good example codes or share your personal experiences? Much appreciated.

  • What you described is exactly what you should do. It may seem as a burden for such a small library, but see it as exercise how to write good tests and test data. And if your library grows it's much easier to extend existing tests and test data. E.g. you just copy an existing test file, change a few things and you have a whole new test. – Simon Jan 9 '19 at 22:13

Testing a parser is tedious but not difficult, because you are dealing with a nice textual input format. What I tend to do is to also define a textual output format (often needed anyway for debugging), and then write examples that consist of an input and an output text. You can place multiple test cases into one file, for example using C++ raw strings, a YAML file, an ad-hoc format, or the Cucumber/Gherkin language. For an INI file, we might have test cases like this:

Feature: simple keys

  Scenario: last key wins
    When I parse the following:
      foo = 42
      bar: 17
      foo = baz
    Then the output is:
      key "bar" value int 17
      key "foo" value string "bar"

  Scenario: key needs separator to value
    When I parse the following:
      foo bar
    Then the error is:
      expected separator at input.txt line 1:
      foo bar
             ^-- here

So we can systematically go through the various syntax features and write tests that demonstrate these features, and we can also write test cases that show that errors are correctly rejected with a helpful error message. Because a single file contains both input and output, keeping them in sync is not that difficult.

These parser tests can be kept distinct from tests of the API that your library provides, for example that the data model can be manipulated or queried with various methods. You also do not have to create a test directory with hundreds of small example files, as its sufficient to test the parser by feeding it a string. You can have a few separate tests that involve file I/O.

  • And then you write a parser to interpret the test files? How do you test that one? – D Drmmr Jan 10 '19 at 18:54
  • @DDrmmr The example in my answer uses Gherkin syntax, which is used by a variety of BDD tools in many languages. These tools only require you to match a string or regex for each step to map it to some action. But I've also written test runners that read tests in an ad hoc format – this often ends up being much simpler. I learned that technique from Brian Kernighan's essay Code testing and its Role in Teaching (2006) where the testing strategy of the awk tool is discussed. – amon Jan 10 '19 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.