0

As part of a black-box testing activity, I would like to check the executable I'm testing generates a binary file that responds to its specifications given a specific input. I'm wondering how I can check that.

I have two solutions in mind but each of them has some drawbacks:

  1. Compare bit by bit the output with a binary of reference. If the two files are equal, the test passes
  2. Parse the generated binary file and check if it contains the expected data

The advantage of the solution 1 is there is no need of an external tool to check the output is correct. However, how to generate the reference? Most of the time, the tested executable will be used to generate the binary of reference. It makes the test pointless because we are testing it against itself (it may have a purpose in regression testing). Another inconvenience I see with this solution is we will need to rewrite all the tests each time a single bit of the output changes.

On the other hand, with the solution 2, we will only need to modify the parser if the specification changes (plus eventually some tests related to this change). However, where does this parser come from? Is it a parser dedicated for testing this binary? In this case, how can we be sure the parser is correct? Or is the parser part of another binary included in our software application? In this case, our tests are dependent on another part of the software. If there is a regression in it, our tests will fail despite the fact the issue is elsewhere.

As you can see, there are pros and cons for both of the solutions and I'm not able to determine which is the best choice in which conditions. Which solution would you choose? Or is there a third solution I have not imagined?

2
  • Regression testing is not pointless. And it is usually not "the tested executable" which creates or created a certain binary file, but a prior version of the current executable under test, so I am not sure what you mean by 1, the description is not consistent.
    – Doc Brown
    Jun 6 '21 at 17:15
  • @DocBrown I agree. By pointless, I meant it doesn't deserve what we want to do here: validating the executable is giving us the right output (and so, from the first version of the executable).
    – Pierre
    Jun 6 '21 at 17:20
1

It depends on what you need to assert.

Another inconvenience I see with this solution is we will need to rewrite all the tests each time a single bit of the output changes.

You're not wrong, but that is an inevitability of testing. Tests are liable to change when the code they test is changed intentionally.

I say intentionally, because part of this process is that you can identify the difference between a refactor (output did not change) and a change (output did change), which means you can vet if your change to the code did or did not change anything about its workings. Both cases are desirable purposes for test suites.

The "problem" here is that you've defined your assertion down to the bit, and therefore the assertion logic is liable to fail when a single bit changes. Had you loosened your assertion logic, that would maybe not have happened.

It all depends on what you need to assert, and that is contextual. For example, if the only thing you needed to test is e.g. if the output file does not contain any negative numbers (when parsed), then you write a test that asserts precisely that. Not a bit-by-bit check, but a parse-and-check-for-negatives assertion.

Then, when your code starts outputting a different file (bitwise) which still doesn't contain any negative numbers, then your test won't fail on the new code, and it won't need to be updated.

This depends on many consideratons:

  • The current use case and precisely what needs to be asserted
  • What can reasonably be asserted and is considered as sufficient confirmation of the test passing.
  • The difficulties in writing the precise assertion logic as opposed to an imperfect shorthand which may be much more performant.
  • The difficulty in having to update this shorthand somewhat more regularly, weighed against the cost of writing the precise assertion logic that requires less overall changes.
  • The cost of being wrong. NASA tests their software much more rigorously than some run-of-the-mill CMS system, for good reason: the consequences of failure in "production" are much more severe. You have to gauge how bad it is for you when a mistake slips through the cracks of an imperfect testing suite, and weigh this against how much work you put into creating the strongest testing suite you can.

Reuse, when confirmed by tests, is acceptable.

However, where does this parser come from? Is it a parser dedicated for testing this binary? In this case, how can we be sure the parser is correct?
In this case, our tests are dependent on another part of the software. If there is a regression in it, our tests will fail despite the fact the issue is elsewhere.

There is a careful balance to be struck here, both trying to not blindly rely on an untested part of your own codebase, but also not having to reinvent the wheel between your codebase and your test suite.

If your codebase already contains the parser itself, because it actually uses it (not just for tests!), then you simply write a test suite for the parser itself. This vets the parser, and makes you able to distinguish between a "real" test failure and a "because the parser is wrong" test failure.

If your codebase does not contain a parser, and you therefore only need it for testing purposes and not purposes of production runs of your code, you build the parser separately, you then test it separately, and then you release it separately. These releases will be confirmed to be working (due to their tests) and therefore you can import this fixed parser (whether as an .exe, package reference, DLL import, ...) without needing to question its correctness.

If your parser is a third party tool, then it's usually okay to rely on it working as intended. However, in cases where you really can't afford such a blind reliance (e.g. life-threatening mistakes due to faulty software), you simply revert back to the original point and write specific tests to confirm that the parser works as expected. If both the actual tests and the parser tests pass, then you have confirmation that both your code and the parser work.

5

Or is there a third solution I have not imagined?

There are two additional solutions.

Snapshot testing

The first one is based on what's called snapshot testing. The idea is that there is, at a given moment, a source code which is manually inspected and found to be correct. At this moment, the snapshot test runs, and generates a snapshot which would be used as a reference the next time the same test runs.

The next time, either the actual output is equal to the reference, or it's not. In the first case, the test is green. In the second case, the test is red, and it belongs to the developer to consider how to make it pass. Either there is an actual regression which needs to be solved, or the change is perfectly legit, and it's up to the reference to be updated (usually through the specific tooling shipped with the framework which runs the snapshot tests).

In general, snapshot testing is used in the domains where the output is humanly readable. One example is React, where HTML code generated by the components can be snapshot-tested: it's not particularly funny to type all the HTML code yourself in an ordinary test, so a snapshot test saves you a lot of time by automating this step. Another example is the tests of the visual interface—would it be a desktop application or a web page—where a screenshot is taken at a given moment, and compared through pdiff with one or multiple references.

In both examples, a human being is generally apt to figure out relatively easily, based on the diff, whether the difference between the current value and the reference is valid or not. In your case, it's complicated. In fact, some diffs of the binary output could easily be recognized as a regression, but other situations could be much more complex. Reusing a parser from the source code within the test could help. If value ≠ reference, the test would show both the binary diff and the diff of the parsed tree.

Serializing the expected object

Another approach is to describe, in your test, the object you're expected to have, and then serialize it to binary in order for it to be compared to the actual value you get from your code.

This assumes that you have a serializer that you can trust, i.e. that passes all the unit tests.

In other words, if the application that you black-test performs a bunch of steps to construct an object, then serializes it, and then flushes it to stdout, your black box test will reuse the part which deals with serialization, and test only the other steps and the chain itself.

Given that the serialization is usually easy to unit test, there shouldn't be too much problems with this technique.

2
  • I’d feel better about upvoting this if “assumed to be correct” was “manually inspected and found to be correct”. I’ve done regression testing this way. We always cracked the snapshot open and picked It apart before blessing it. Otherwise all you prove is that it’s just as buggy as it’s always been. Jun 6 '21 at 19:18
  • @candied_orange: valid point. Answer fixed. Jun 9 '21 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.