Let's say I want to add tests to a software that has some flaws/bugs.
I want to add the tests before I fix the bugs. And I want to actually merge them in the main branch, not just have them in a feature branch or bugfix branch somewhere.
(EDIT: See below as to why I might want to do this.)
If this is relevant, I am working with phpunit. But I am looking for more general answers.
I see the following options:
- Create failing tests that describe the desired behavior, and merge those into the main branch. This means any QA pipeline has to be suppressed.
- Create passing tests that describe the current (flawed) behavior, and merge those into the main branch. Add
@todocomments if possible. Update them later as part of the bug fix.
- Use a fancy mechanism that allows switching between "desired" mode and "current" mode. Running the tests in "desired" mode would make them fail, but running them in "current" mode would make them pass.
- Don't merge the tests, until I fix them.
Personally, for my own projects, I like options 2 and 3.
But when working in a team, I want to be able to justify my strategies with online references, instead of just personal opinion.
So, is there any named "best practice" or "pattern" that matches option 2 or 3 above? Or am I missing something, and there are even better strategies available?
EDIT: Why merge the tests before fixing the bug?
(I am adding this section to avoid having to put all of it into the comments.)
Why would I want to merge the tests into the main branch, before the bug fix? This has been a contention in the comments, so I am summarizing possible reasons here.
Possible reasons why we can't fix the bug now:
- The bug fix might be risky or disruptive, perhaps we want to postpone it for a new major version.
- The bug fix, unlike the test, needs dedicated effort by the maintainer, whereas the new test can be developed by a contributor.
- There is no budget / resources to fix the bug now.
- The existing behavior is not fully understood, and we don't really understand the implications that fixing it would have.
- We know the current behavior is wrong, but we don't know have precise spec for the desired behavior. Perhaps we need feedback from the client or management.
- The bug has been discovered while doing something else, and fixing it now is out of scope.
So why would we want to merge the tests anyway?
(assuming option 2 from above)
- The tests cover more than just this one bug.
- We want to reward and credit contributors who provide tests, even before fixing a bug.
- We want to provide a starting point for contributors who attempt to fix the bug.
- We want a git diff of the fix to give a precise overview of behavior change.
- We want to detect when the bug might be fixed as a side effect of other changes.
- We want to detect when the wrong behavior changes to a different flavor of wrongness, as a side effect of other changes.
- If a user of the software package discovers something that looks broken, they can look into the tests to find the behavior documented, and marked as a "known issue", with link to the issue queue.
- The git history will reveal more information about the difference in behavior before and after the fix. It will also reveal in which version the problem was fixed.