In my test environment we are running the same set of integration (and sometimes unit) tests against different systems that have different capabilities.
Some of the tests cannot be executed against some of the systems because a feature that enables this scenario is missing or we need completely different behavior.
Currently we are using MSTest and have tests with this pattern:

public void TestMethod1()
    if(targetSystem.SupportsFeatureX == false)
        Assert.Inconclusive("Not supported by underlying system");
    // the code for the actual test follows

Should we mark the test as Inconclusive or just let it Pass if the feature check is not satisfied?
Please note that targetSystem is a third party system that we have no control over so we just live with features that are present.

  • 1
    do you expect FeatureX to be supported? if yes, then test should fail. If no, then test should pass – gnat Mar 4 '14 at 12:30
  • TestMethod1 is targeting a few different targetSystems that support FeatureX. I am not testing if FeatureX is available or not. I am testing if my code behaves properly in case FeatureX is there or just skip the test otherwise – ViktorZ Mar 4 '14 at 12:40

If your test framework supports it, you should mark tests that can't be executed due to missing (external) dependencies as skipped. That gives the clearest picture in the overview.

If the test framework does not support a 'skipped' status, then 'inconclusive' is a next-best option. At least, 'inconclusive' won't give as much of a misleading result.

Using 'pass' for tests that were not actually executed can give a misleading impression regarding the stability of a feature. Just imagine the questions you would get if the statistics show that the tests for Feature X consistently pass on systems A, B and C, but regularly fail on systems Y and Z. Is there a problem specific to systems Y and Z or do systems A, B and C not have feature X and is there a generic problem in the feature?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.