Is it bad practice to compare string representation on an object instead of its properties during unit tests?

Question

Given a class as follows:

class IntList {
public:
    IntList(std::vector<int> list) : m_list(list) {}

    std::vector<int> list() { return m_list; }

    std::string toString() {
        std::string repr;
        for (const auto integer: m_list) {
            repr += "int:" + std::to_string(integer) + ";";
        }
        return repr;
    }
    
private:
    std::vector<int> m_list;
};

Now I have an unrelated function initializeIntegers() whose I want to test the returned value. Is it acceptable during unit tests to use toString() to compare the expected values of the object?

void testCase() {
    const auto intList = initializeIntegers();
    assert((intList.toString() == "int:0;int:11;"));
}

It seems weird to me, I would rather compare the value returned by list() instead. However, I can't find any reference stating whether or not using the string representation is OK. Am I wrong if I suggest to not rely on string representation during unit tests? Do you know of any authoritative references explaining it?

Note that both list() and toString() methods exist and are used regardless of the unit tests (they weren't added just to be able to test the class).

Answer 1

Let's say you check whether two values are equal. Your check can go wrong in two ways:

• The values are equal, but the string representation is not (this is unlikely).
• The values are different, but the string representation is the same (say the values are 0.048 and 0.052 and both are represented as the string "0.05").

The second misses a test failure, and worse, it misses a subtle test failure that you won't pick up otherwise. The first case is reasonably harmless - it's annoying because you have a test that fails but shouldn't so you fix the test.

Where it gets really bad is if your test should compare three values a, b, c and you totally missed that your text representation only contains a and b. Now c is completely untested.

So you need some very, very good unit tests for toString() first if you want to rely on it in a unit test.

Answer 2

The main disadvantage of comparing string representations is that now there are two different reasons why that test might start failing:

• the functionality of the class was broken by a bad commit
• the toString() functionality was changed by an unrelated commit that is fine by itself (perhaps the new toString() is a lot more readable!)

...and it's not obvious to tell which it is at a glance.

How great that drawback is depends on how likely you are to evolve your string representations.

Answer 3

Is it bad practice to compare string representation on an object instead of its properties during unit tests?

It's perfectly ok.

When you compare string representations you are testing:

• toString(),
• everything toString() uses to create the string representation, and
• not necessarily any properties at all

toString() might use properties. Might use something else. All we learn from the test is if whatever it uses works.

The way unit tests work (the good ones anyway) they don't force you to use a particular implementation. So a 'string representation' test is blissfully unaware of whether properties exist at all.

So it's perfectly ok. So long as you don't care whether properties exist.

Answer 4

Assuming that you have already unit tested the IntList class, and know that all its public methods work correctly, then it's up to you how you test your new function. Using the toString() method of your class seems like perfectly good thing to do.

When it comes down to it, unit tests are written by programmers to convince themselves that the code they are writing works. Formal acceptance tests are a different thing.

Answer 5

In many cases, objects may have aspects of their state which cannot be meaningfully captured within a string representation. Suppose there is a structure type foo which for simplicity simply contains an integer, and another structure type bar which for simplicity simply contains a foo*. There also exists an instance of foo, called boz, which happens to hold the number 12.

There is no way for code which produce a string representation of a bar, but which is unaware of the existence of boz, to meaningfully distinguish between a bar that contains a pointer to boz, and a bar that contains a pointer to some other foo that happens to contain the number 12. If, however, the value stored in boz were to change, the effect of that on a bar would depend upon whether that bar pointed to boz.

For some kinds of objects, such distinctions wouldn't matter, but for other types of object the relationships among pointers contained therein may be far more important than any information held in the targets of those pointers, but cannot be meaningfully be expressed as a string.

Answer 6

For completeness, I'm just adding a few thoughts I had trying to articulate why it didn't seem as good practice to me:

• This question only exists for one reason: the lack of an == operator implemented in IntList. If such operator existed, I would not have to chose between toString() or list() comparison. If I had to define this operator, the implementation would more likely be based on the two objects properties rather than their string representation.
• The string representation is mainly intended for humans. If I decide to improve the string representation to ease debugging, I don't want unrelated test cases to break because of this. The software internally uses the properties, those are the one I want to check to prevent regression in the code.
• The toString() methods are often more complicated than properties lookup. It's transforming the object to another representation, which may loose information (like integer rounding) or miss internal bugs. I remember we used to rely on debugString() method of Protobuf, but null values weren't displayed which wasn't very convenient during tests. We should not rely on implementation details like so.
• Looking at other languages, many Python classes provide __repr__ and __str__ methods out-of-the-box, yet they are rarely relied upon for unit tests.
• Finally, the choice comes down to knowing what constitutes the identity of the object. If the string is a perfect representation of what the object is, then it is maybe acceptable to use it for comparison.

Answer 7

Most answers disapprove assertions against strings, as of my time of writing. But, I want to say something for the other side.

In essence, to test against strings, is to test against a Domain Specific Language (DSL).

This is a strong tool to use when testing within the normal scope feels no longer handy. For example, you wanted to test a series of dependent events happened in order, instead of mocking quite a few objects and testing that many functions did get called, you could write something like this:

expected_execution_log = """
    Change highlight threshold 50 -> 70
    Update label value 50 -> 70
    Request new batch of data (threshold = 70)
    Set grid overlay None -> Loading
    Received new batch of data (threshold = 70, count = 2)
    Set grid content (count = 2)
    Set grid overlay Loading -> None
"""

assert log == expected_execution_log

This pedagogical example illustrates user interaction, async execution, view updates, and content updates. It's very hard to describe the behavior completely in your test.

Taking a step back, if you assert against execution logs, you only have to insert a spy logger, and add some log statements

You should have already got accustomed to logging, with or without tests.

Always remember that you are never able to describe the full set of the behavior of your application, because doing so equals to writing your application again, just with more clumsy syntax and more verifications^[1]. The reason why we love tests so much is that by probability, if you accidentally, insanely, messed up with something, you are likely to trigger the trap that your tests are holding to, so that the sober, sensible you would get alarmed^[2] and fix that problem.

So, you should remember that by testing, especially by test driving, you are actually setting traps for a future, careless you, to trigger that trap. If the trap itself is so difficult for you to setup, then why not use a weaker, yet simpler one? After all, setting traps is dangerous too! A convoluted test is not that important than a clear test. Just write the feel and it's OK to live with it.

---

[1] In this world, we can only know things through their behaviors, or phenomena. If A is able to replicate every possible behavior of B, then A effectively contains B. Since this is not a philosophy lecture, I prefer not to elaborate on this point.

[2] You might be thinking: there is still a small chance that I don't get alerted! Yes. You might have guessed it. A possibility of 100% of getting alarmed when anything goes wrong? We call this static verification. There are many possible approaches, such as symbolic execution, which aim to verify the program before it gets put to usage. This goal is usually pursued for critical software on cars, planes, civil infrastructure, space exploration and military utilities.

Answer 8

A unit test usually consists of 3 parts

// arrange (any required set up / initialization before the actual test )

//act (the action that is being tested)

// assert (where the expected result is compared to the actual result)

The fact or facts that you assert are entirely up to you as the developer, there is no best practice on whether you should test a single property, multiple properties, a string representation of an object, how many times a method is called, the type of exception thrown, the exception message, the state of multiple objects or even whether it is Tuesday when the test finishes.

That is all up to you as the developer. You have to decide what represents a passing test and what doesn’t.

Remember, a unit test is confirmation that the unit of code under test will behave in the expected manner — only you can decide what you expect it to do.

Answer 9

One wrinkle I'd consider is that some languages will print out its string representation with some randomness included - for example, a dictionary or hash might print out its keys and values in random order, or in the order they happened to be inserted, or some other algorithm. Comparing these sorts of objects by looking at their string representation would result in random failures, which are the worst kind. I've been bitten by this before when trying to compare JSON strings.

This doesn't apply to the specific case quoted in the code, but is something to consider for the more general question posed.