1

Context

While trying to get my parser classes under test, I noticed a common challenge for (unit-)testing them: They have only one public method, with a string as input and the parsed class as output. They are further divided into methods for readability, but those are all private. These classes are not huge - I would say on average we're talking about the whole class being 150 lines of code (including whitespace) with around 8 private methods.

According to this question, you shouldn't make methods public just for the sake of testing. I don't have the same problem as that question, because calling that one public method does indeed touch all the private methods.

Issue

However, I struggle a bit with the thought of re-creating the whole string input with just a slight change for every if in those private methods. Also these tests seem like integration-, rather than unit-tests to me, simply because they test too many lines of code at once - I'm not sure about this though.

Possible options I thought of

I also see some options which could be of help with that. Lets go with an example of a parser that needs an XML-formatted input string. I could use a hard-coded XML string or I could dynamically create that XML with language-given XML tools (e.g. XDocument in C#).

If we further think about a program that saves some XML-data in a file to read it later, that program has the corresponding "encoder" for that parser. I could create the XML with the encoder and decode it with the parser/decoder, effectively testing that they work together - which seems like all the program needs anyway, unless the spec says otherwise.

Question

Are parsers a special case in unit testing?
If yes, to what extent? Are their certain "dos and don'ts" to look out for which differ from "standard" testing?
Or am I maybe just writing my parsers in a very inconvenient way?

  • 3
    The difference between integration testing and unit testing is not the number of lines of code that are tested, it's the number of units that are tested. A typical unit test verifies the behavior of one unit (e.g., one class) in isolation. An integration test verifies the behavior of many units working together. – Solomon Slow May 2 '18 at 14:42
5

Parsers are not a special case, what you desribe is somewhat common. If the class in question is of a reasonable size, adheres to seperation of concerns, single-responsibility principle and so on, then you should test it through its single public method. This just means the class has a tight public surface which is a good thing.

But you should not focus on "testing every if" in the class. That would be coupling the test to the implementaion which we want to avoid. You should test all the specific requirements which the parser is supposed to satisfy.

Also these tests seem like integration-, rather than unit-tests to me, simply because they test too many lines of code at once - I'm not sure about this though.

No, the distinction between unit test and integration test is whether the test touches seperate subsystems. If it is all in single class, then it is definitely not an integration test.

If we further think about a program that saves some XML-data in a file to read it later, that program has the corresponding "encoder" for that parser. I could create the XML with the encoder and decode it with the parser/decoder, effectively testing that they work together - which seems like all the program needs anyway, unless the spec says otherwise.

It makes sense to test a "roundtrip" to ensure the components work together. This is a valuable form of integration test. But it is important that you also unittest input/output of the components individually, since otherwise you run the risk of both components having a similar bug (not unlikely if written by the same person) which whould not be discovered by the integration test.

Your test input/output samples should be at the same abstraction level as the class you are testing. E.g if the class takes strings as input, you should test it with hardcoded strings. Otherwise your input generation gets to complex and you run the risk of bugs in input generation which may in turn mask bugs in the class you are testing. Tests should be simple code.

  • I like this answer, especially the last paragraph makes a lot of sense. However, I am still on the fence about "testing every if", either way. I concur that one shouldn't test the implementation, but every if in the parser can possibly change the result and as thus probably should be tested. Or, thinking the other way around: If I did TDD, there would be no if that isn't being tested. – R. Schmitz May 2 '18 at 15:21
  • @R.Schmitz: Oh I agree, it is more about the mindset you should have when testing. You should test against expected behavior (i.e. the requirements), not against the structure of the code. But since every `if`` presumably is there for a reason, every branch will be tested eventually, if you test against the expected behavior. – JacquesB May 2 '18 at 16:30
3

Parsers are no different than other classes with few public methods. Therefore, the thing to do is the same as with other such classes: if your class C uses many, many internal methods c1..c99, then almost certainly many of these can be grouped into related clusters of functionality that serve the higher purpose of parsing.

These methods should become public methods of new classes, and then you can unit-test them in the unit tests for the new classes D, E, F and G.

  • Well, I am not talking about crazy god-ish classes here - Lets say 8 private methods, the whole class being 150 loc. I'll add that information to the question. – R. Schmitz May 2 '18 at 12:51
2

Developing Unit Tests requires some change in typical coding behavior and thought.

At it's base level, a unit test is a set of inputs and a set of expected results. You provide the inputs in your unit test, and you assert against the expected results. The more granular you can break this down the better your unit tests are at detecting failures. You should design your unit tests as granular as you feel has value. Getting high "code coverage" is not nearly as important as testing inputs and expected outputs, as if you cover the latter well you will cover the former well (as well as getting all the in-class interactions that may not be well understood by coding for code coverage).

As for how to provide inputs, there's absolutely nothing wrong with hard coding values into code for unit tests. We avoid doing magic strings and magic numbers in regular code because it usually has little/no context, and we like to have constants in one easy to find place to update at some future point. With unit tests, you should have a clearly defined test name that defines the context (I almost always go with MethodName_InputState_ExpectedResults, and frameworks like X Unit allow you to defined string names for tests that can be different from method names), and if you have a failing unit test, Visual Studio and other IDEs allow you to jump to that unit test (or a global search for the test name can find it very fast). I wouldn't worry about coming up with a way of avoiding hard-coded strings (unless you're trying to do a test of your parser from random input or input sets, in which case some testing frameworks have tools for that).

2

Unit tests don't care about private classes, because unit tests inherently don't care about the processing step. That is to say, unit tests don't care about parts of the processing.

A unit test is inherently only interested in the input and output. As you said, there's only one method with takes a string input and outputs a class. This means that you can't make a unit test smaller than the conversion from the string to the class.


As a somewhat silly example to prove the boundaries of what a unit test is and isn't interested in:

If you put raw chicken in a working oven, then you get cooked chicken. Otherwise, the oven is defective.

What the unit test cares about:

  • When I put raw chicken in the oven, cooked chicken comes out of it (test success)
  • When I put raw chicken in the oven, something that is not cooked chicken comes out of it (test failure)
  • When I put raw chicken in the oven, the oven explodes (fatal test failure)

What the unit test doesn't care about:

  • What happens when you put nothing in the oven.
  • What happens when you put raw beef in the oven.
  • What happens when you put a live chicken in the oven.
  • How the Oven turns raw chicken into cooked chicken.
    • The oven might heat the chicken meat.
    • The oven might grill the chicken meat.
    • The oven might teleport the raw chicken to the nearest restaurant, have it cooked, and then teleport the cooked chicken back into the oven.
    • The oven might simply disintegrate the raw chicken and create cooked chicken out of thin air.

All of these ovens would pass the unit test I mentioned. Because the unit test only cares that the oven yields cooked chicken. It does not care how the oven came by the cooked chicken.

Notice how the unit test doesn't care about what happens when you put other foods in the oven. The test description is only interest in the output when the specified input is given.


Also these tests seem like integration-, rather than unit-tests to me, simply because they test too many lines of code at once - I'm not sure about this though.

The distinction between unit tests and integration tests has nothing to do with how many lines of code are being tested.

Unit tests test the logic in the smallest chunks possible.

Integration tests test whether multiple components are working together.

  • If you were testing the inverse method (class to string), that would be a unit test.
  • If you were testing the inverse method, including if the file was written to disk (class to file with the string as content), that would be an integration test.

However, I struggle a bit with the thought of re-creating the whole string input with just a slight change for every if in those private methods.

Nothing is preventing you from creating a reusable testing approach. Grossly oversimplified for the sake of example, let's say you have three sections (A, B and C), each section can be one of two options (1 and 2).

private string BuildString(int sectionA, int sectionB, int sectionC)
{
    return $"{sectionA}{sectionB}{sectionC}";
}

This can simplify your testing code:

[TestMethod]
public void Test_CanParse_111()
{
    var inputstring = BuildString(1,1,1);

    var expectedOutput = new MyClass() { A = 1, B = 1, C = 1 };   

    var receivedOutput = myParser.Parse(inputstring);

    Assert.AreEqual(expectedOutput.A, receivedOutput.A);   
    Assert.AreEqual(expectedOutput.B, receivedOutput.B);   
    Assert.AreEqual(expectedOutput.C, receivedOutput.C);
}

This simplifies the testing logic as much as it can, while still making sure that the test itself has complete control over the input that is used.

Keep in mind that the unit test shouldn't care how the input string is created. It only cares that the input string always accurately reflects the case that it's testing for.


Looking at the above example, it seems reasonable that A, B and C are separate pieces of logic. One does not influence the other.

If that is the case, then you can write your tests to reflect that.

  • A test to see if A = 1 is handled correctly.
  • A test to see if A = 2 is handled correctly.
  • A test to see if B = 1 is handled correctly.
  • A test to see if B = 2 is handled correctly.
  • A test to see if C = 1 is handled correctly.
  • A test to see if C = 2 is handled correctly.

You don't need to test every combination, you just need to test every individual outcome. If the value of one does not affect the other, there's no point in testing both of them.

For my example, this lowers the amount of tests needed from 8 to 6. That's not a significant change. However, I assume that your input string has more than three properties, each of which can contain more than two possible values. The reduction in amount of tests will significantly increase if there are more properties and possible values.

However, I can see some people on disagreeing on this point, instead suggesting to still test every combination. I disagree with that (I consider it testing beyond reason), but in order to sidestep that discussion altogether, let me offer a second way of simplifying the testing logic:


AutoFixture can be used to randomize your input (with the option of defining reasonable boundaries.

This forgoes a rigid "testing every case" approach, in favor of a "test sampling" approach. Every time the test is run, you get a random value (unless you've specified a constant value you want to be used).

The benefit here is that you don't have to manually write out every combination. It also allows you to test for values that you wouldn't have thought to test for.

The drawback is that you no longer have executive control over the input values. Instead of knowing which value is being used to test, you instead know that e.g. a number between 0 and 9 was used. This obviously also means that you need to be able to discern this value in the output. If that int value causes multiple fields to all be influenced in varied ways, then you should not be testing with reasonably randomized values.

Note that AutoFixture can go hand in hand with the above method. Consider the first test:

  • A test to see if A = 1 is handled correctly.

This could be handles by having AutoFixture create your input string with A = 1 (hardcoded), but having B and C randomly generated by AutoFixture.


I also see some options which could be of help with that. Lets go with an example of a parser that needs an XML-formatted input string. I could use a hard-coded XML string or I could dynamically create that XML with language-given XML tools (e.g. XDocument in C#).

As per my example above, both are valid approach. Like I said, the unit test shouldn't care how the string was created. It only needs to measure its expectation of the output based on the input string.

Keep in mind here that you're not testing whether your code can generate correct XML. You're testing that your parser can parse XML (that is guaranteed to be structurally correct.
If you make a mistake in the XML generation logic, that should still be visible in the test result (parsing errors); but that doesn't mean that XML validation is the actual intention of the test.

If your XML generation is less than trivial, it wouldn't be the worst idea to first unit test the XML generation itself, before allowing the XML generator to be used in the subsequent parser test. But this gets into a multi-tiered testing approach which is well beyond the initial scope of the question.


If we further think about a program that saves some XML-data in a file to read it later, that program has the corresponding "encoder" for that parser. I could create the XML with the encoder and decode it with the parser/decoder, effectively testing that they work together - which seems like all the program needs anyway, unless the spec says otherwise.

First of all, if you get the file from disk, it's an integration test, not a unit test.

Secondly, I may be missing the point here, but what is the benefit you're hoping to get from using a file from disk instead of an in-memory string? Whether you create that string in a file, or in your IDE, seems to be the same amount of work for the same amount of testing validation.

  • "Integration tests test whether multiple components are working together." - when refactoring the class into multiple classes, doesn't this make a unit test an integration test then ? – Steve Chamaillard May 2 '18 at 14:34
  • @SteveChamaillard: Depends on the refactoring. Refactoring in the sense that you've split the workload over two separate layers; yes it becomes an integration test. Refactoring in the sense that you create additional internal classes, then it remains a unit test as you're still testing the smallest possible chunk. In short, unit testing is seeing if Bob can throw a ball and catch a ball; and seeing if Tom can throw a ball, and catch a ball. Integration testing is seeing if Tom and Bob can play catch. – Flater May 2 '18 at 14:37
  • "First of all, if you get the file from disk, it's an integration test, not a unit test." Well, classes called Encoder and Parser probably shouldn't do any IO. I just wanted to express that the application has code to both encode and decode the information. I currently have a lot of parsers that only decode data from a server, while the application has no idea how to encode it. – R. Schmitz May 2 '18 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.