This feature allows you to write and commit a test separately from the code that will make it pass.
(It can be compared to double-entry bookkeeping in accounting that allows one to exchange goods and payments at separate moments while keeping a record of the outstanding debt in the meantime.)
A test suite should always succeed. But what if we have tests for features that are not yet implemented, or tests for known bugs that we can't fix right now?
We write an xfail test.
The xfail tests encodes our assumptions. It is subject to source control, it must still compile, and we can execute it. This prevents our assumptions from falling out of date with the current state of the software. But during normal work with the test suite, these xfail tests do not cause the test suite to fail. This is similar to a skipped test, except that when an usually-skipped test is executed and fails, that would fail the complete test suite. Failure of an xfail test does not fail the test suite.
In test runners without xfail tests, we instead have to rely on the number of passing tests, or the percentage of passing tests. But this is extremely fragile. First, these counts are wrong if new tests are added – which should happen all the time. Second, these accumulated metrics fail to account for the case where one tests starts passing but another test starts failing.
In contrast, xfail tests ensure that passing tests keep passing, and will not hide accidental regressions. This works like a ratchet: our test suite can't slip back and introduce regressions, but can proceed forward by removing the xfail annotation from a test as soon as it passes.
Xfail tests as implemented in pytest are distinct from tests that are expected to produce an error. The xfail test should pass and is expected to pass in the future, but is expected to fail given the current state of the software. In contrast, tests that are expected to produce an error fail if there is no such error. Such error tests are e.g. useful to assert that input validation succeeds. Most test frameworks have special annotations to catch an expected exception.