Is there a benefit to unit testing code whose sole purpose is to generate non-human-readable code in another language?

Question

A significant portion of the application I work on every day consists of Javascript that outputs a lot of (what might as well be) Excel spreadsheet formulas. Yes, Excel is barely a Turing-complete language, but the formulas are more than complex enough to qualify as "code". They're not meant to be human-readable or human-editable. The users never see these formulas at all, only the values which those formulas evaluate to.

The main problem with our codebase right now is a lack of automated testing, and thus far I've focused my efforts on integration tests which run both the Javascript and the formulas, then assert on the final values. After all, we don't care what formulas were used to create those values, so I don't really see the point in a unit test that asserts on the formulas produced by each piece of Javascript. I'd also expect such unit tests to be exceedingly brittle, as most of our changes to the Javascript code are intended to change the formulas in some way (albeit in a way that shouldn't affect most of the final values).

Note that the code which evaluates the formulas is owned by a different team, and they have loads of unit tests on it. I'm interested in tests for our code, the part that generates the formulas.

Also note that speed is a non-issue; the integration tests I've written so far are so fast they feel more like unit tests.

For the reasons given above I'm convinced that improving our integration test coverage is the higher priority, at least for now, but I honestly can't see any benefit at all to unit testing this code, which feels like a sign that I might be missing something important.

So, is there any purpose to having unit tests as well as integration tests for this particular kind of code, where the sole objective is to generate "code" in some other language, and nobody cares what "intermediate code" is used to achieve the final result?

Answer 1

I would generally argue that there are uses. Those uses might not apply to you in this specific scenario, but they might.

• What happens when your integration tests fail?

Now you get to dig into the code to narrow down what the actual root cause was. If you had unit tests for just the formula generation, it would provide clear evidence (without any work or guessing on your part) if it was the formula generation that broke or the execution bits.

• What happens when you move to a new execution framework?

Presumably, your company will still need to calculate all of these values. If your execution framework changes, you're going to need to change your output syntax. Ideally you have some implementation agnostic form for your formulae. Unit tests help unsure that part of the code is right so that when you change your target syntax you can have confidence that you're not changing the semantics as well.

• What happens when the execution framework is updated? Does it break your tests? How do you know that is what broke the tests?

Decoupling makes for less impact of changes, even in tests.

• How long does it take?

Maybe not a problem for you, but integration tests tend to be more complex. That means slower to run, slower to write, slower to maintain. If unit tests are 95% as effective as the integration tests, but take half the time to make, they may be the better option (at least in part).

Again, these may not apply to you in this situation, but they would be my default arguments for unit testing even in an environment where integration tests seem abundantly sufficient.

Answer 2

A shorter form of your question would be "does unit tests make sense for a compiler". I believe the answer to that question is a strong yes. You need to know that the smallest parts of your transformation pipeline do what they should. You need to know that they're combined correctly and so on.

I'd say that unit tests, integration tests and end-to-end tests all make sense.