4

Suppose we have a project consisting of many microservices, all of which use a common library. The common library has been put into a separate git repository, and each microservice is also in its own individual git repository.

When the time comes to make a change to the common library, how should that be done? Because all of the microservices use it, it seems like it would be necessary to clone all the microservice repositories that were not cloned locally already, update each of them to point to the new version of the common library, publish locally the new version of the common library, and then run all of their tests. And then, in principle, this has to be done on the CI system as well, because otherwise there could be a subtle difference in the local environment that makes the change happen to build OK on the local environment, but not in CI!

If we don't do this, but simply do the lazy thing and update only the common library and the particular microservice we are working on at the moment, we run the risk that we accidentally break something in another microservice and that this only becomes apparent later when the dependency on the common library gets bumped in the latter microservice. If the common library were an open source project and the microservices depending on it were third party code, we could just say "tough luck - you fix it on your side, or raise a PR to fix it on our side. It's not our responsibility to babysit your repositories." But since they are our repositories, they are our responsibility - so we shouldn't really break them gratuitously with a poorly-thought-out change to the common library.

However, the approach of testing all common library changes everywhere that I have outlined is laborious - and doesn't scale particularly well, either. (Imagine if Google had their billions of lines of in-house code, not in a mono-repo but in this kind of setup - how would they be able to make safe changes to shared libraries in a scalable way?)

What sorts of approaches can be used to better manage such updates and make the QA more efficient?

13
  • Is the common library in its own repo because it has value on its own or because it's shared between the services? Is there ever a case when the common library will change outside the context of what it provides to the services?
    – D M
    Commented Apr 5, 2021 at 14:52
  • 1
    @RobertHarvey the external API of the shared library isn't supposed to change every time we build it, no. But it might change accidentally - e.g. due to a subtle change in behaviour which doesn't cause any of its own unit tests to fail. Commented Apr 5, 2021 at 15:01
  • 1
    And your integration tests are supposed to catch these subtle changes in behaviors? Commented Apr 5, 2021 at 15:08
  • 1
    How long does it take to run your entire body of unit tests and integration tests for all of your microservices? Commented Apr 5, 2021 at 15:21
  • 1
    @Calabacin that's all very well in theory, but in practice a library's own tests don't always catch all problems. Commented Jun 1, 2021 at 9:08

1 Answer 1

2

In my company we solve this problem simply by uploading the shared libraries to Nexus. Here is how we do it:

  1. Library has its own integration and release jobs.
  2. When we create a new release of the library it is uploaded to Nexus with a new version number.
  3. All applications that use this library can simply specify what version they need and download it automatically as part of your build process (using maven, gradle, nuget, shell script, etc.).
  4. Each application can use a different version of the library, and they can upgrade at their own pace.

Nexus is an 'artifact repository server', but you could use your own storage solution if you don't have one.

I hope this helps you.

3
  • 1
    This approach works... until it doesn't. Consider the following corner case: a refactoring is done to the library, which is intended to be safe and only affect the internal implementation, not the API. Since no-one needs this refactoring, no-one upgrades to the new version of the library. Subsequently, an urgent requirement comes in and a change needs to be made to the library and all microservices updated to use the new version. Then the incompatibility/brokenness is discovered. Commented May 2, 2021 at 9:12
  • @RobinGreen you solve this using major and minor versions and proper documentation. Anyone upgrading to a new major release should expect potential incompatible changes. Also versions have a lifetime during which they should receive updates to urgent issues (i.e. bugs). If a project needs a new feature in my library and I make it, surely they'll understand they will need to use the latest version. In my opinion none of the problems you mention are related to our way of working, as they would all happen in any other methodology you apply.
    – Calabacin
    Commented May 31, 2021 at 17:23
  • Also it is common that a library fixes issues and bugs in their library, hence "only affecting internal implementation". Projects should get these fixes or at least be made aware of consequences of not doing it via release notes. I really cannot think of any such change that wouldn't be caught with proper unit and integration tests.
    – Calabacin
    Commented May 31, 2021 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.