Some test frameworks, notably DejaGnu and py.test, have an "XFAIL" flag/status for a test which says that this test is expected to fail. If the test succeeds, it's marked as "XPASS" (unexpected pass).

Isn't this just a redundant overcomplication? It seems much easier and comprehensible to just use an assert that would expect an error -- if the error happens, the test passes, if it doesn't, it fails.

gcc in particular is having so much trouble with this status it makes test output all but useless. I was ready to write this feature off as yet another misguided quirk of history -- until I saw it in py.test.

What's the purpose of this feature? What can it do that SKIP and asserts cannot?

  • 4
    There is very specific guidance about XFAIL on the py.test page you linked: why it exists, how it is used and under what conditions it is useful. What more are you expecting from us? – Robert Harvey Dec 5 '18 at 13:55
  • @RobertHarvey I've written in bold what I'm expecting. pytest guidance says how it should be used but doesn't say why it should be used over a regular test. – ivan_pozdeev Dec 5 '18 at 14:06

A test suite should always succeed. But what if we have tests for features that are not yet implemented, or tests for known bugs that we can't fix right now?

We write an xfail test.

The xfail tests encodes our assumptions. It is subject to source control, it must still compile, and we can execute it. This prevents our assumptions from falling out of date with the current state of the software. But during normal work with the test suite, these xfail tests do not cause the test suite to fail. This is similar to a skipped test, except that when an usually-skipped test is executed and fails, that would fail the complete test suite. Failure of an xfail test does not fail the test suite.

In test runners without xfail tests, we instead have to rely on the number of passing tests, or the percentage of passing tests. But this is extremely fragile. First, these counts are wrong if new tests are added – which should happen all the time. Second, these accumulated metrics fail to account for the case where one tests starts passing but another test starts failing.

In contrast, xfail tests ensure that passing tests keep passing, and will not hide accidental regressions. This works like a ratchet: our test suite can't slip back and introduce regressions, but can proceed forward by removing the xfail annotation from a test as soon as it passes.

Xfail tests as implemented in pytest are distinct from tests that are expected to produce an error. The xfail test should pass and is expected to pass in the future, but is expected to fail given the current state of the software. In contrast, tests that are expected to produce an error fail if there is no such error. Such error tests are e.g. useful to assert that input validation succeeds. Most test frameworks have special annotations to catch an expected exception.

  • "In test runners without xfail tests, we instead have to rely on the number of passing tests, or the percentage of passing tests." -- huh? I always thought it only matters if all the tests pass or not. Code either works as expected, or it doesn't. There's nothing in between. When fixing a bug, a failing test is written in a topic branch, and it's only committed to a main branch together with a bugfix, after it starts passing. – ivan_pozdeev Dec 5 '18 at 14:16
  • @ivan_pozdeev yes but the whole premise of xfail tests is that we have tests that cannot pass yet due to an external constraint, e.g. unimplemented features or known bugs. It is not always feasible to keep those tests on a separate branch as the tests might be xfail for a long time (potentially forever), so the branch would drift away from the master branch. Keeping xfail tests in the master branch ensures that they can still be run. – amon Dec 5 '18 at 14:19
  • "We write an xfail test." No. We mark them as ignored with appropriate comment. – Euphoric Dec 5 '18 at 14:20
  • 1
    @ivan_pozdeev It's not intended to replace an issue tracker. It's just a convenient way to keep not-yet-passing tests around. This also won't inflate any test results. There's a difference between an xfail test failing as expected, and a normal test passing. This is super convenient when implementing a spec with an official test suite. For example, a test suite with 60% passing and 40% xfail tests suggests we have implemented ~60% of the spec. BDD style tests are similar, as one user story is tackled at a time (and each story equates to a test). – amon Dec 5 '18 at 15:14
  • 2
    @ivan_pozdeev Except now a not-yet-passing test is documented and explicitly supposed to fail. If it starts passing, that's marked as a failure since it's xfail. Which means you have a failing test that must be addressed (whether it's due to actually implementing/fixing or accidental) - unmarked as xfail (since you now expect it to pass since you intentionally fixed) or refactored (if you accidentally fixed one case of it). The test isn't unmaintained, in fact it forces you to maintain your expecting-failure tests (unlike commented tests marked as ignore). – Delioth Dec 5 '18 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.