Should I write and commit unit tests for minor changes and bug fixes?

Question

For example, if the change is "return users' full names instead of just last names", is it worth it to add a test for it? Would it make the test suit fragmented and confusing?

Context: My team got handed an existing code base to extend and maintain. It currently has unit tests for basic and important features. We'll start the work with some bug fixes and minor changes. I'm new to unit testing and want to learn the best practice. Some posts seem to suggest I should write tests for everything. Is it true or is it a case-by-case basis?

Answer 1

"return users' full names instead of just last names"

This implies a necessary application behavior (because if it didn't, who told you this needed to be changed?), which in turn is something that should always be backed by tests. For the purpose of the answer, I'm solely focusing on unit tests as the kind of testing isn't the focus here.

You should've always had a test to confirm this application behavior.

is it worth it to add a test for it?

Okay, turns out you didn't already have a test even though the (prior) application behavior existed.

What is it you're proposing here? They did it wrong in the past (by not backing this by tests), so I won't bother fixing it either? The absence of a test therefore invalidates the need for a test?

Whether or not a test already exists for this is irrelevant with regards to the question of whether a test is warranted here.

Would it make the test suit fragmented?

How would testing an application behavior "fragment" your test suite?

Would it make the test suit confusing?

If a test to check for a specific return value would be confusing, the task would be to learn how to write better tests, not to reconsider having tests in the first place.

Overall, I disagree with the underlying "I'm not confident with this yet, so I just won't do it" undertone that the question seems to be based on. That just perpetuates the inexperience with the given topic.

Answer 2

Some posts seem to suggest I should write tests for everything. Is it true or is it a case-by-case basis?

This question potentially leads to an over-simplification or at least a way to miss the point of testing, as it could be framed in such a way that it might sound like approaching testing simply for its own sake or following what others tell you to do.

Consider re-framing this about what kinds of tests should exist, and how to avoid the trap of producing tests whose purpose is simply about checking a box or meeting some coverage statistic.

What's missing is consideration about to whether a test has any real value to anybody; in particular, whether the tests are checking real functionality/behaviour that users, QA testers and stakeholders care about. Given that the question mentions bug fixing, that sounds to me a lot like something these people should care about.

On the other hand, it's quite easy to write totally meaningless tests to feed into some arbitrary statistics, by only pointlessly testing the structure of the code, or test behaviour belonging to 3rd-party libraries, or even test the tools/language itself. Most of the time, those kinds of tests take up time without ever serving anybody.

Instead consider the real reason for writing tests in the first place; particularly thinking about how people (developers, users, QA testers, other stakeholders) might get real value out of them.

In general, tests are a way of protecting the quality of the software - their value to users is fewer bugs up-front and fewer regression issues in the future when the software changes. Their value to QA testers and stakeholders is similar, as well as confidence that developers have understood the requirements and are doing their job properly. Finally, their value to developers is helping those developers avoid breaking existing code, and have confidence in their understanding of how the code works and why the code exists.

With that in mind, consider that the existence of a bug must imply that either some test/s had been written with wrong expectations, or a testing gap; otherwise somebody else would have noticed the bug before. It also suggests some oversight by whoever touched the code before (because all developers are only human).

Lastly, tests can indeed be a helpful tool for organising your thought processes and double-checking your assumptions; even if they are a bit like "marking your own homework", they're just one layer in (hopefully) many other layers of quality assurance, protecting against the existing behaviour being broken by future developers.

If there are correct tests around some behaviour, then future developers (or 'future you') can confidently revisit that same code; perhaps to adapt it to some new requirements, or maybe to refactor it, or perhaps to fix some other bug; all without fear of accidentally breaking something which was working before they touched it.

As an added bonus, future developers can also read and run/debug tests in their IDE to help them understand why some code exists, understand its expectations and the original requirements, so the tests serve as documentation for them too -- except unlike plain-English written documentation, this 'documentation' is always verified directly against the real code before anything is merged into the main branch, so it cannot go stale.

Answer 3

In this specific case, yes, definitely.

In general the default position should be that if you fix a bug, you add a test case which fails before you fix the bug, and succeeds after you fix it.

There are some situations where I would depart from this. One example is improving error messages: if the bug is that an error message is misleading I will try to improve the message, but I won't add a test that the product produces this exact message, because having lots of such tests actually makes it harder to improve the product in future.

Another case is where it is extremely difficult to reproduce the circumstances of the error, and where I have a high level of confidence that the fix is correct. An example might be a multithreading problem that is fixed by synchronizing a particular method. Sometimes in such cases it's more productive to add tests that are broader in scope, e.g. improving the testing of multithreading in general, rather than writing a test designed to catch the specific bug.

Answer 4

Did you have a function returning the full name, together with a unit test, before you started? If not then you added a new function and a unit test for it.

Did you modify a function returning all kinds of data about a customer? That should have unit tests. And the unit tests should currently be failing. So you need to modify them. And check any code that calls this method.

If any object with this information is returned, don’t change the meaning of the object, but change it so that code assuming the old meaning doesn’t compile anymore and gets fixed.

Answer 5

The scarce of details about the in test application restraint the possible answers to a single answer, write unit tests for every piece of behaviour that has to be preserved when replacing the in use technologies at the system boundaries the way the presentation component and the persistence component are.

Existence of bugs implies the unit test coverage is shallow and it should be improved by testing untested scenarios. With a comprehensive unit test suit bugs shouldn't exist.

For a 100% web application the unit tests might not exist and yet have a safe to change code base by having comprehensive user interface test ran by cross-browser tests frameworks... spoiler alert: ad following... the way Selenium WebDriver framework is.

If it would be to choose an approach I would look for a solution to test the application just through its graphical interface and skip the unit tests development.

Answer 6

Testing isn't a philosophical or religious faith position. It's an engineering step to get confidence in your product. This is true for every branch of engineering.

How much testing should you do? The answer, always, is how much you care about releasing a product with bugs. The more testing you do, the fewer bugs are left in the product.

The first and best option is always reviewing. Let's assume that happens. Then...

Best practise generally is to start with system-level tests. Does your product as a whole do what it's supposed to? This doesn't just catch bugs in your implementation, but also errors in your requirements which don't give you the product you actually need. Evidence is that this gives best value for money in testing. This is rarely scriptable though.

After that, you have module-level tests. Does each coherent block do what it's supposed to? And after that at the lowest level, you have unit tests which check each function does what the design said it was supposed to.

Unit tests generally cost the most effort for the least benefit. However if you have a piece of functionality where you can't easily system-level test it, unit testing lets you exercise those cases in isolation much better. Unless you're doing safety-related engineering where this is mandated, you'll get more value for money by being smart and picking where to do that.

This is based on years of safety-related software, as a summary of development standards from Ford, MISRA and DO-178B, and using one of our projects at the time as a case study in which V&V phases found how many issues of what severity for how much effort spent in that phase.

Answer 7

Bugfixes should not need new tests, they should at most need fixes to existing tests. Simply because the fixed code should already have been tested, indicating that the tests were also faulty!

Changes to functionality either require new tests and/or changes to existing tests.

And keep in mind that 100% test coverage is nice in theory, but in practice may not be worth the effort (and in cases may be impossible to attain). Also remember that a bad test is worse than no test as it gives a false sense of security.

Answer 8

The described scenario is in my mind not a bug fix. You either:

• modify the return value of an existing function, or
• add a new function

Regardless, this should be tested somehow. Exactly how and where you do the test depends on a lot of factors so neither "never do unit tests" or "always do unit tests" are correct. You need to have a test strategy defined in the project and some kind of framework for actually running the tests. One very simple strategy is to require every function to have a unit test. This comes with costs at several levels. One cost is how long the tests take to run. Another cost is the sense of false security that might arise.

In this scenario there will always be new edge cases that the old function did not support. Edge cases are in my experience good places to test in unit tests. Assuming, say, you read first and last name from a database:

• user not found in database
• no first name in the database
• no last name in the database
• neither first or last name in database
• a very long first name in the database *)
• a very long last name in the database *)
• the resulting name becomes very long *)

*) There are always limits on the largest values that can be held in internal variables and on the size of return values. Trying to enter a too large value in an integer / too long string could end up in problems such as memory corruption surfacing a lot later.