3

In an environment where you can't merge code that fails tests you obviously can't merge new, failing tests.

But how to deal with a newly found bug exploit, meaning you wrote a new test exposing an already existing bug? Test-driven-development would say: Hooray, you have the test, now fix the bug and then commit both.

But what if you a) are a Test engineer who found the exploit but are not able to fix it or b) you would be able to fix it but not right now?

Option One: Commit the test to an extra branch, mention it in the bug issue and essentially leave the code until somebody has mercy and bases their fix upon this branch and commits both.

Option Two: Mark the test somehow as "not nice that it fails but expected", merge it and the Continuous Integration lets it trough. As soon as the bug is fixed, you end the test's special treatment and mark it mandatory. Maybe, the CI would even display the list of expected failures so you have an additional kind of bug tracker.

Are there any common practices / standards out there? Especially for Option Two, do standard test frameworks like JUnit, Google Test,... have any way of modeling it?

Improve this question
6
  • Sounds like a team decision to me, which comes down to team goals and what processes you want. Personally, I like the idea of committing the failing test to master with a reference to an open ticket. The business can prioritize that bugfix in whichever iteration they deem fit. This would also give future coders visibility of it when they work in the code. But that's just me given the processes I typically work within.
    – bitsoflogic
    Aug 22, 2018 at 19:10
  • @bitsoflogic: In Continuous Integration environments, checking in a broken build to the master branch can cause all manner of undesirable side-effects.
    – Robert Harvey
    Aug 22, 2018 at 19:32
  • @RobertHarvey Great point! Of course, depending on the severity of the issue, those effects may be desired.
    – bitsoflogic
    Aug 22, 2018 at 19:42
  • A smart CI system has separate steps for builds and tests, which can easily support adding new, even failing tests - a simple baseline comparison can automatically ignore tests that never passed before - after all CI's ultimate goal is to detect regressions in the code quality and these new failing tests are not actually regressions. IMHO TDD would have a really tough time with a CI system which doesn't have such capability.
    – Dan Cornilescu
    Aug 23, 2018 at 1:42
  • 1
    One would be Zuul, there are a few pointers in this answer: softwareengineering.stackexchange.com/a/377315/187812. Another would be ApartCI, which I created, sorry for the advertising ;)
    – Dan Cornilescu
    Aug 23, 2018 at 9:35

2 Answers 2

Reset to default
4

Never break the build. But tests that should pass in the future are completely valid. Many test runners understand an “xfail”, “pending”, or “todo” category for this kind of case. If one of those tests (unexpectedly) starts passing you can remove this category. By merging these todo tests, you can make sure that they continue to compile and run, which may be preferable over storing these tests separately.

Examples:

  • In JUnit you can use annotations to mark pending tests, though you might need a separate library. Simply ignoring the test might be insufficient because the test won't be attempted.
  • Google Test can mark tests as disabled and can execute those tests, but interprets a failure as a failure rather than pending. With enough macros, you could roll your own solution.
  • A pending state is completely normal in BDD tools like Cucumber.
  • Pytest has excellent support for an xfail category.
  • Perl's Test::More has TODO blocks.

Whether to use this approach depends a lot on your team's workflow. Todo tests are great when you are implementing a spec or are tackling a backlog of user stories – especially in a BDD context where the test is the specification. And todo tests can be good for highly important problems where the test is valuable documentation.

But for normal tests, it might be better to not bog down the test suite with lots of failures. When writing an issue or bug report, consider including the test code in that issue rather than merging it. Writing a good test is already a large part of debugging and solving the problem, so ideally the problem can be fixed quickly.

Improve this answer
1
  • "you can make sure that they continue to compile and run" would tip the scale for me personally in favor of option two. As @Bryan-Oakley said, this is a subjective choice, though
    – PhilLab
    Sep 7, 2018 at 16:20
5

Option one. Just because you can't merge code doesn't mean you can't check it in. You simply need to create a branch for your failing test, and then it's ready when someone starts to work on it.

That being said, this isn't for us to decide. This is something your team needs to decide for itself. There are no magic bullets that you can adopt that will make this situation easier. Do what works best for your team and your way of working.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.