10

Given a class as follows:

class IntList {
public:
    IntList(std::vector<int> list) : m_list(list) {}

    std::vector<int> list() { return m_list; }

    std::string toString() {
        std::string repr;
        for (const auto integer: m_list) {
            repr += "int:" + std::to_string(integer) + ";";
        }
        return repr;
    }
    
private:
    std::vector<int> m_list;
};

Now I have an unrelated function initializeIntegers() whose I want to test the returned value. Is it acceptable during unit tests to use toString() to compare the expected values of the object?

void testCase() {
    const auto intList = initializeIntegers();
    assert((intList.toString() == "int:0;int:11;"));
}

It seems weird to me, I would rather compare the value returned by list() instead. However, I can't find any reference stating whether or not using the string representation is OK. Am I wrong if I suggest to not rely on string representation during unit tests? Do you know of any authoritative references explaining it?

Note that both list() and toString() methods exist and are used regardless of the unit tests (they weren't added just to be able to test the class).

5
  • 5
    "I would rather compare the value returned by list() instead." - then your test would depend on the correct implementation of list() instead of the correct implementation of toString(). And, not to forget, on the correct implementation of ==. Not much of a difference.
    – Bergi
    Jun 22, 2022 at 21:03
  • Do you have any sensible assertion framework to use instead? If you don't, I'd expect such case to be a workaround for the lack of proper tooling, like GoogleTest matchers. If the alternative is to write EXPECT_THAT(initializeIntegers(), ElementsAre(0, 11)), I'd say that using toString() is a horrible choice. If the alternative is to write a for loop, I'd say it's a good choice to pick a concise option, even if it's semantically inferior.
    – Frax
    Jun 23, 2022 at 9:56
  • 3
    if your goal is to test for equality, then use the == operator. In your example it is not implemented, but your unit test need already shows that there is a need for this operator. I would add the operator and use that in your test. Implementing the == operator forces you to define when you consider two objects to be the same, which is not necessarily thought about when making a toString() method.
    – slingeraap
    Jun 23, 2022 at 14:04
  • Yes, comparing string representations can let some errors squeak through. But it is far better to test that way than not to test at all. And better to have tests that are easy to write, that allow you to re-factor and change your implementation as required, then to have invested a lot of time of code used only for testing, not for the actual application… If that investment then becomes an obstacle to re-factoring and improving your design. If writing the non-string compare tests is fast And easy enough, sure do that from the start. We call it agile for a reason.
    – Krazy Glew
    Jun 24, 2022 at 3:20
  • string(x)=string(y) only implies x=y when string(x)=string(y) implies x=y. That's when string returns a canonical/normal form--by definition of canonical/normal form.
    – philipxy
    Jun 24, 2022 at 22:41

9 Answers 9

32

Let's say you check whether two values are equal. Your check can go wrong in two ways:

  • The values are equal, but the string representation is not (this is unlikely).
  • The values are different, but the string representation is the same (say the values are 0.048 and 0.052 and both are represented as the string "0.05").

The second misses a test failure, and worse, it misses a subtle test failure that you won't pick up otherwise. The first case is reasonably harmless - it's annoying because you have a test that fails but shouldn't so you fix the test.

Where it gets really bad is if your test should compare three values a, b, c and you totally missed that your text representation only contains a and b. Now c is completely untested.

So you need some very, very good unit tests for toString() first if you want to rely on it in a unit test.

8
  • 2
    I have, on occasion, been lazy enough to use JSON serialization as a shortcut, e.g., JObject.FromObject(a).ToString(), but I then usually, still compare to a hard-coded "expected" string rather than serializing another object.
    – Corrodias
    Jun 22, 2022 at 21:42
  • 16
    "The values are equal, but the string representation is not" could very well happen if you're implementing an unordered collection like a dictionary or set. Jun 23, 2022 at 4:21
  • 1
    1024 could be 1,024 or 1’024 or 1_024 or 1K or $400.
    – gnasher729
    Jun 23, 2022 at 21:26
  • 1
    @cmaster-reinstatemonica In Java, at least, this is not correct. The API says: "In general, the toString method returns a string that "textually represents" this object. The result should be a concise but informative representation that is easy for a person to read." A full representation of a large array would not be concise. Jun 30, 2022 at 7:39
  • 1
    cmaster, Apple violated an expectation that was completely unjustified. People used a method that was meant to produce output to send to the debugger.
    – gnasher729
    Jul 6, 2022 at 7:12
14

The main disadvantage of comparing string representations is that now there are two different reasons why that test might start failing:

  • the functionality of the class was broken by a bad commit
  • the toString() functionality was changed by an unrelated commit that is fine by itself (perhaps the new toString() is a lot more readable!)

...and it's not obvious to tell which it is at a glance.

How great that drawback is depends on how likely you are to evolve your string representations.

2
  • 3
    Third reason: there are elements of the object that are not included in the output of ToString().
    – bta
    Jun 22, 2022 at 23:12
  • A false test failure is no big deal, you just fix it. And it answers the question “is it bad practice…”. If it makes your tests fail, then yes, it is bad practice :-)
    – gnasher729
    Jul 6, 2022 at 7:14
3

Is it bad practice to compare string representation on an object instead of its properties during unit tests?

It's perfectly ok.

When you compare string representations you are testing:

  • toString(),
  • everything toString() uses to create the string representation, and
  • not necessarily any properties at all

toString() might use properties. Might use something else. All we learn from the test is if whatever it uses works.

The way unit tests work (the good ones anyway) they don't force you to use a particular implementation. So a 'string representation' test is blissfully unaware of whether properties exist at all.

So it's perfectly ok. So long as you don't care whether properties exist.

1
  • 2
    In TDD, testing by comparing string representations is not that unusual, overall when you are in the early stages of the test. It's quite useful because the opposite might lead you to add useless getters or break encapsulation for "the sake of the test".
    – Laiv
    Jun 23, 2022 at 10:41
1

Assuming that you have already unit tested the IntList class, and know that all its public methods work correctly, then it's up to you how you test your new function. Using the toString() method of your class seems like perfectly good thing to do.

When it comes down to it, unit tests are written by programmers to convince themselves that the code they are writing works. Formal acceptance tests are a different thing.

1
  • 1
    I want to test the output of initializeIntegers(), I'm not testing the IntList class itself.
    – Delgan
    Jun 22, 2022 at 10:14
1

In many cases, objects may have aspects of their state which cannot be meaningfully captured within a string representation. Suppose there is a structure type foo which for simplicity simply contains an integer, and another structure type bar which for simplicity simply contains a foo*. There also exists an instance of foo, called boz, which happens to hold the number 12.

There is no way for code which produce a string representation of a bar, but which is unaware of the existence of boz, to meaningfully distinguish between a bar that contains a pointer to boz, and a bar that contains a pointer to some other foo that happens to contain the number 12. If, however, the value stored in boz were to change, the effect of that on a bar would depend upon whether that bar pointed to boz.

For some kinds of objects, such distinctions wouldn't matter, but for other types of object the relationships among pointers contained therein may be far more important than any information held in the targets of those pointers, but cannot be meaningfully be expressed as a string.

7
  • 1
    Why couldn't pointers be expressed as strings? Java's default toString method does basically that.
    – Bergi
    Jun 22, 2022 at 20:59
  • @Bergi: Java's default hashCode() method, used by toString(), is designed so that references to the same object will always yield the same value, and references to different objects will usually yield different value, but there is no guarantee that to completely unrelated objects won't have the same hash code, and the odds of that happening are not even terribly remote if many objects' hash codes are observed shortly after the objects are created. Further, there is no way code can know what the "correct" hash code (or default toString() representation) of an object should be.
    – supercat
    Jun 22, 2022 at 21:13
  • @Bergi: An object reference contains, as part of its implied encapsulated state, the whereabouts of every single other reference to that object that exists anywhere in the universe. If one were to simultaneously change every reference to an object so that they all identified a different object whose fields contained the same contents, that would not affect the state encapsulated by the object, but it's impossible to tell if running two different programs to the same point would leave an object in the same state without knowing the whereabouts of every reference to it.
    – supercat
    Jun 22, 2022 at 21:16
  • Sorry for mentioning Java, you're right there… But I don't get your other point. In unit testing, we do not care about the entire universe of objects. If we care about an object's identity, we can stringify its memory address, if we only are about structural equality, we can stringify its content values. (If we care about the object graph, its a bit more complicated, but even that can be serialised into a string). Either way, the string captures all the information that we care about, and can be used for comparisons in testing.
    – Bergi
    Jun 22, 2022 at 21:26
  • @Bergi: The fact that a particular object happens to be at address 0x08675309 on one particular program execution says nothing meaningful about whether address 0x08675309 will identify that same object on any future program execution.
    – supercat
    Jun 22, 2022 at 21:45
1

For completeness, I'm just adding a few thoughts I had trying to articulate why it didn't seem as good practice to me:

  • This question only exists for one reason: the lack of an == operator implemented in IntList. If such operator existed, I would not have to chose between toString() or list() comparison. If I had to define this operator, the implementation would more likely be based on the two objects properties rather than their string representation.
  • The string representation is mainly intended for humans. If I decide to improve the string representation to ease debugging, I don't want unrelated test cases to break because of this. The software internally uses the properties, those are the one I want to check to prevent regression in the code.
  • The toString() methods are often more complicated than properties lookup. It's transforming the object to another representation, which may loose information (like integer rounding) or miss internal bugs. I remember we used to rely on debugString() method of Protobuf, but null values weren't displayed which wasn't very convenient during tests. We should not rely on implementation details like so.
  • Looking at other languages, many Python classes provide __repr__ and __str__ methods out-of-the-box, yet they are rarely relied upon for unit tests.
  • Finally, the choice comes down to knowing what constitutes the identity of the object. If the string is a perfect representation of what the object is, then it is maybe acceptable to use it for comparison.
1

Most answers disapprove assertions against strings, as of my time of writing. But, I want to say something for the other side.

In essence, to test against strings, is to test against a Domain Specific Language (DSL).

This is a strong tool to use when testing within the normal scope feels no longer handy. For example, you wanted to test a series of dependent events happened in order, instead of mocking quite a few objects and testing that many functions did get called, you could write something like this:

expected_execution_log = """
    Change highlight threshold 50 -> 70
    Update label value 50 -> 70
    Request new batch of data (threshold = 70)
    Set grid overlay None -> Loading
    Received new batch of data (threshold = 70, count = 2)
    Set grid content (count = 2)
    Set grid overlay Loading -> None
"""

assert log == expected_execution_log

This pedagogical example illustrates user interaction, async execution, view updates, and content updates. It's very hard to describe the behavior completely in your test.

Taking a step back, if you assert against execution logs, you only have to insert a spy logger, and add some log statements

You should have already got accustomed to logging, with or without tests.

Always remember that you are never able to describe the full set of the behavior of your application, because doing so equals to writing your application again, just with more clumsy syntax and more verifications[1]. The reason why we love tests so much is that by probability, if you accidentally, insanely, messed up with something, you are likely to trigger the trap that your tests are holding to, so that the sober, sensible you would get alarmed[2] and fix that problem.

So, you should remember that by testing, especially by test driving, you are actually setting traps for a future, careless you, to trigger that trap. If the trap itself is so difficult for you to setup, then why not use a weaker, yet simpler one? After all, setting traps is dangerous too! A convoluted test is not that important than a clear test. Just write the feel and it's OK to live with it.


[1] In this world, we can only know things through their behaviors, or phenomena. If A is able to replicate every possible behavior of B, then A effectively contains B. Since this is not a philosophy lecture, I prefer not to elaborate on this point.

[2] You might be thinking: there is still a small chance that I don't get alerted! Yes. You might have guessed it. A possibility of 100% of getting alarmed when anything goes wrong? We call this static verification. There are many possible approaches, such as symbolic execution, which aim to verify the program before it gets put to usage. This goal is usually pursued for critical software on cars, planes, civil infrastructure, space exploration and military utilities.

0

A unit test usually consists of 3 parts

// arrange (any required set up / initialization before the actual test )

//act (the action that is being tested)

// assert (where the expected result is compared to the actual result)

The fact or facts that you assert are entirely up to you as the developer, there is no best practice on whether you should test a single property, multiple properties, a string representation of an object, how many times a method is called, the type of exception thrown, the exception message, the state of multiple objects or even whether it is Tuesday when the test finishes.

That is all up to you as the developer. You have to decide what represents a passing test and what doesn’t.

Remember, a unit test is confirmation that the unit of code under test will behave in the expected manner — only you can decide what you expect it to do.

-2

One wrinkle I'd consider is that some languages will print out its string representation with some randomness included - for example, a dictionary or hash might print out its keys and values in random order, or in the order they happened to be inserted, or some other algorithm. Comparing these sorts of objects by looking at their string representation would result in random failures, which are the worst kind. I've been bitten by this before when trying to compare JSON strings.

This doesn't apply to the specific case quoted in the code, but is something to consider for the more general question posed.

1
  • For someone creating JSON for example it would be a great option to create deterministic output. So object X converted to JSON would always produce identical JSON data. A stronger option would be trying to make it so that small changes in X produce very similar JSON. So using a deterministic but pseudo-random hash table while producing the JSON would be no good.
    – gnasher729
    Jul 18, 2023 at 16:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.