9

Some test frameworks, notably DejaGnu and py.test, have an "XFAIL" flag/status for a test which says that this test is expected to fail. If the test succeeds, it's marked as "XPASS" (unexpected pass).

Isn't this just a redundant overcomplication? It seems much easier and comprehensible to just use an assert that would expect an error -- if the error happens, the test passes, if it doesn't, it fails.

gcc in particular is having so much trouble with this status it makes test output all but useless. I was ready to write this feature off as yet another misguided quirk of history -- until I saw it in py.test.

What's the purpose of this feature? What can it do that SKIP and asserts cannot?

Improve this question
2
  • 7
    There is very specific guidance about XFAIL on the py.test page you linked: why it exists, how it is used and under what conditions it is useful. What more are you expecting from us?
    – Robert Harvey
    Commented Dec 5, 2018 at 13:55
  • @RobertHarvey I've written in bold what I'm expecting. pytest guidance says how it should be used but doesn't say why it should be used over a regular test.
    – ivan_pozdeev
    Commented Dec 5, 2018 at 14:06

2 Answers 2

Reset to default
16

This feature allows you to write and commit a test separately from the code that will make it pass.
(It can be compared to double-entry bookkeeping in accounting that allows one to exchange goods and payments at separate moments while keeping a record of the outstanding debt in the meantime.)

A test suite should always succeed. But what if we have tests for features that are not yet implemented, or tests for known bugs that we can't fix right now?

We write an xfail test.

The xfail tests encodes our assumptions. It is subject to source control, it must still compile, and we can execute it. This prevents our assumptions from falling out of date with the current state of the software. But during normal work with the test suite, these xfail tests do not cause the test suite to fail. This is similar to a skipped test, except that when an usually-skipped test is executed and fails, that would fail the complete test suite. Failure of an xfail test does not fail the test suite.

In test runners without xfail tests, we instead have to rely on the number of passing tests, or the percentage of passing tests. But this is extremely fragile. First, these counts are wrong if new tests are added – which should happen all the time. Second, these accumulated metrics fail to account for the case where one tests starts passing but another test starts failing.

In contrast, xfail tests ensure that passing tests keep passing, and will not hide accidental regressions. This works like a ratchet: our test suite can't slip back and introduce regressions, but can proceed forward by removing the xfail annotation from a test as soon as it passes.

Xfail tests as implemented in pytest are distinct from tests that are expected to produce an error. The xfail test should pass and is expected to pass in the future, but is expected to fail given the current state of the software. In contrast, tests that are expected to produce an error fail if there is no such error. Such error tests are e.g. useful to assert that input validation succeeds. Most test frameworks have special annotations to catch an expected exception.

Improve this answer
9
  • "In test runners without xfail tests, we instead have to rely on the number of passing tests, or the percentage of passing tests." -- huh? I always thought it only matters if all the tests pass or not. Code either works as expected, or it doesn't. There's nothing in between. When fixing a bug, a failing test is written in a topic branch, and it's only committed to a main branch together with a bugfix, after it starts passing.
    – ivan_pozdeev
    Commented Dec 5, 2018 at 14:16
  • 1
    @ivan_pozdeev yes but the whole premise of xfail tests is that we have tests that cannot pass yet due to an external constraint, e.g. unimplemented features or known bugs. It is not always feasible to keep those tests on a separate branch as the tests might be xfail for a long time (potentially forever), so the branch would drift away from the master branch. Keeping xfail tests in the master branch ensures that they can still be run.
    – amon
    Commented Dec 5, 2018 at 14:19
  • "We write an xfail test." No. We mark them as ignored with appropriate comment.
    – Euphoric
    Commented Dec 5, 2018 at 14:20
  • 3
    @ivan_pozdeev It's not intended to replace an issue tracker. It's just a convenient way to keep not-yet-passing tests around. This also won't inflate any test results. There's a difference between an xfail test failing as expected, and a normal test passing. This is super convenient when implementing a spec with an official test suite. For example, a test suite with 60% passing and 40% xfail tests suggests we have implemented ~60% of the spec. BDD style tests are similar, as one user story is tackled at a time (and each story equates to a test).
    – amon
    Commented Dec 5, 2018 at 15:14
  • 6
    @ivan_pozdeev Except now a not-yet-passing test is documented and explicitly supposed to fail. If it starts passing, that's marked as a failure since it's xfail. Which means you have a failing test that must be addressed (whether it's due to actually implementing/fixing or accidental) - unmarked as xfail (since you now expect it to pass since you intentionally fixed) or refactored (if you accidentally fixed one case of it). The test isn't unmaintained, in fact it forces you to maintain your expecting-failure tests (unlike commented tests marked as ignore).
    – Delioth
    Commented Dec 5, 2018 at 21:42
1

They let the test suite work while still allowing expected errors

When you write code, and you test it with tests, it doesn't matter whether the test passes or fails, it matters whether the result was what you expected. Let's say my product is in development and I compile it with a debug flag set. I write a test to make sure that debug mode is off during production, but I'm okay with debug mode being on for my development version. When I run tests on my development version, it will fail because debug mode is on, but I'm okay with that so I set all the development version debug tests as XFAIL tests. This means that the test is expected to fail so if it does fail, it's okay. If the test passes, it's an XPASS or unexpected pass, because debug mode is off and that's not what's expected. Even though the test passed, it's not expected, so it's not good. One person could want debug mode on and another wouldn't. They both have different expectations and the test I wrote doesn't satisfy everyone's expectations for the software. The test passing doesn't mean the software is good. This reinforces the fact that a passing test isn't a good one, but rather a test with an expected result is a good one. This is why the config script for most software creates a Makefile with some XFAIL tests so that the suite doesn't fail for dependency issues where some optional software isn't present, or debatable features, like debug mode.

What about if the issue is fixed?

Let's say that a developer writes code that has features that rely on the tree library. The tree library's newest version doesn't work with our developer's code, so for all the development versions, the features are disabled and the test that figures out if the tree library works is set as XFAIL in the config script, because the tree library doesn't work, and that's expected. Now, what if the tree library's developer fixes the library so now it works with our developer's code? Our developer would just set the test as normal and re-enable the feature and it would be normal again. This is the same for if we just don't install the tree library. As soon as we compile the software with the tree library, the config script will set the test as normal because the tree library is correctly installed. What if we don't want the tree library and the code's tree library-dependent features? That's okay because we can usually pass a flag to the config script that tells it to ignore the fact that the tree library doesn't exist. The config script will disable the features and set the test as XFAIL. The tree library doesn't exist, the test fails, but it's okay, because it's XFAIL and it's expected.

Improve this answer
2
  • 1
    The first example looks like something for a conditional skip, or, even better, for commits that are different between branches (git allows to do this). You are going to edit tests each time you compile the project with different flags, really?
    – ivan_pozdeev
    Commented Jul 3, 2020 at 18:44
  • It's just an example, I don't know much about git, so I didn't know what a developer would do for different branches of the software.
    – Foxler2010
    Commented Jul 7, 2020 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.