9

I recently started using Semantic Versioning.

Due to conflicts with my previous versioning convention (which I created myself), I am very confused about versioning unit tests as part of the public API.

As mentioned in the SemVer documentation:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backward compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated. It MAY be incremented if substantial new functionality or improvements are introduced within the private code. It MAY include patch level changes. Patch version MUST be reset to 0 when minor version is incremented.

I believe that unit tests are a part of the public API as they can have huge effects not only on the quality of the source code, but also on the functionality of the public API.

What should we do with our versioning system taking into account the importance and influence of unit tests?

How should they affect versioning?

Should they be considered in versioning as a segment of the public API?

Improve this question
6
  • 76
    "I ceartainly believe that unit tests ARE a part of the public API" You will need to expand upon this. Because it makes no sense to me. I think you might be mixing up some things and getting confused.
    – Euphoric
    Commented Feb 11 at 15:22
  • 58
    "I […] believe that unit tests ARE a part of the public API as they can have […] effects on […] the functionality of the public API." – How? You can literally delete every single test and the API will not change one bit.
    – Jörg W Mittag
    Commented Feb 11 at 16:16
  • @JörgWMittag: The parts you omitted make it sound like OP is thinking that the lack of tests negatively impacts the API in the long run, and assuming that this is part of the kind of thing that versioning is meant to protect you from. It's not, but I can sort of see the train of thought there.
    – Flater
    Commented Feb 11 at 22:40
  • 9
    I think that this question is unclear and not useful because it rests entirely on a false premise ("unit tests are a part of the public API"). Now that you're aware that unit tests are not a part of the public API, do you still have a question? If so, can you edit this post to clarify what your question is?
    – Sophie Swett
    Commented Feb 12 at 3:29
  • 10
    @TannerSwett: there is nothing in wrong in asking a question which presents a false premise as long as the OP asks if that premise is really correct and does not insist on it.
    – Doc Brown
    Commented Feb 12 at 9:18

3 Answers 3

Reset to default
60

Your unit tests are not part of your public API. By definition, the API is a software interface that allows two or more systems, applications, programs, or components to communicate. The existence or lack thereof of your unit tests has no bearing on the communication between elements. Your public API and its functionality is what it is, regardless of the state of the tests. That functionality also includes defects or deviations that your tests may catch.

How you treat your versioning depends on your build and release process.

Your major, minor, and patch version numbers are associated with a release. If you are performing a new release, you should update the version number, even if the release only affects your unit tests. Consistent with the Semantic Versioning guidance for dependencies, I would increment the patch version if you did not change anything besides the unit tests. If your unit tests discover a bug and you also fix that bug, depending on the nature of the fix and what you need to communicate to consumers, you may need to increment the major or minor version number.

However, I would consider it unusual to release just because you added unit tests. Semantic Versioning has allowances for build metadata. If you aren't releasing, I would expect this build metadata to track new commits and builds that haven't yet been released. You can combine this with pre-release versioning if you are also targeting new features and haven't settled on compatibility issues yet.

Improve this answer
9
  • 7
    tl;dr: "If you[r unit tests discover a bug and you also] fix that bug ... you may need to increment the [...] version" - this is the answer
    – Cireo
    Commented Feb 12 at 6:14
  • 2
    Why would you do a release if all you've done is change tests would be my question
    – ScottishTapWater
    Commented Feb 13 at 12:12
  • @ScottishTapWater As I mentioned in the first sentence of the last paragraph, I would consider that very unusual, but it does depend on your release process. Perhaps you automatically create a "release" (such as a GitHub Release) for every commit. I would expect build metadata to update, or perhaps a pre-release version to be created, and not a final release.
    – Thomas Owens ♦
    Commented Feb 13 at 13:03
  • To be clear Thom, I know you acknowledge it, I just think that calling it unusual is a bit soft, it's a generally good answer though and I have upvoted it
    – ScottishTapWater
    Commented Feb 13 at 14:16
  • Unit tests aren't a part of the public API. If your release includes the unit tests what you're releasing isn't an API. It's a code base. If you want to version that fine. But stop going on about the API because this aint that.
    – candied_orange
    Commented Feb 14 at 18:37
16

Unit tests provide examples for the semantics of an API, hence they can be seen as part of the API's documentation. So when a change in a unit test shows a change of the API's semantics, this can give the impression the changed unit test should trigger a bump of the version number.

However, that impression is wrong, since it mixes up cause and effect:

  • when the semantics of a component's API changes, then the component itself must have been changed, which causes the version number to be increased.

  • the necessary change of the unit test is merely a consequence of this.

  • when you add or change a unit test without changing anything else in the component's API, then you either fix a bug in the unit tests, or you document a certain behaviour which was already in the component.

Of course, when your distribution package includes all of the unit tests, and only the unit tests have changed from one release to the next, the distribution package should be given a new patch version number, as Thomas Owens wrote. This makes it clear that the distribution package contains changed content, still the API has not been changed.

Improve this answer
5

Unit tests should be a different piece of software from your application and hence not affect the versioning at all.

Having said that, You would normally compile them all at the same time, and you should bump the version number every time you build. So I would increase the patch number (or use a 4th digit) every time the build pipeline runs, which it would for any code change to anything in the build, including adding unit tests

Improve this answer
8
  • 1
    In what sense should the unit tests be a different piece of software from the application? I think we generally want software to be "self-testing" which means it includes its own tests.
    – bdsl
    Commented Feb 11 at 21:07
  • 7
    @bdsl the tests should not be reachable in a production deployment, the test code has a different entry point
    – Caleth
    Commented Feb 11 at 21:41
  • 4
    @bdsl: The test suite and the API are different things. Depending on how you perform your tests, either the test suite and the API both consume the underlying layers; or the test suite consumes the API. In either case, your deploy artifact will be the API's build, not that of the test suite.
    – Flater
    Commented Feb 11 at 22:43
  • 4
    @bdsl: most commercial software does not ship with tests
    – whatsisname
    Commented Feb 11 at 22:49
  • 2
    you dont say what language you are using, but normally you will have you application, and then the unit tests go in a different/project/assembly/app which has a dynamic link to the main app. You then run the tests via a test runner application. When you publish your main app there is no need to "strip" anything, because its binaries dont include the tests, while the test binaries do include the app.
    – Ewan
    Commented Feb 12 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.