5

IDLs like Protobuf, flatbuffers, Cap'n Proto or Thrift allow communication over standardised interfaces between otherwise independent projects.

Most often, these projects will be developed in separate repositories, and while they all require the files generated from the interface definitions, none of them are the clear source for those interfaces.

So then, where should the interface definitions reside? How does that influence building/versioning?

My idea was that the IDL files would sit in a separate repository, but what then? Do I make that a submodule of every repo that needs them? Do I clone them alongside the others and then include them when needed? Where do I put the generated files, should they be committed with each of the client repos?

5

The interface definitions are a dependency of all projects using that interface. How you manage that dependency depends on the nature of the project.

E.g. if the interface definitions are shared between loosely related projects then the interface definitions should have clearly versioned releases. The generated code may be specified as a dependency via your language's package manager. In particular, this package may be a high-level library for using your interface rather than just low-level bindings. Where possible, this is the ideal solution.

If the IDL does not have clear releases, if all projects are evolved together (by the same team), or if you need to regenerate the bindings for each dependent project anyway (e.g. because they use different languages), then including the IDL via git submodule is a decent, language-agnostic solution. You will have to integrate this into your build process manually, and prevent any version conflicts externally. A Submodule pins a specific commit of the included repo. If you want to update the interface description, you have to manually pull and commit the new version for the submodule in each dependent project. Note that some build tools may need extra configuration for submodules, especially if the submodule's origin is not available publicly.

Cloning the interface description separately may work, but means that your projects are no longer tied to a specific interface version which makes debugging more difficult: dependencies ought to be explicit. On the other hand, updating the interface description is easier: pull the cloned repo, then rebuild all dependencies.

You should not commit generated files. That makes it unnecessarily difficult to regenerate the bindings if you change the interface definitions:

  • The diff will contain both the actual changes and the changes in the generated files.
  • The generated files might be edited manually.
  • The IDL might be edited without regenerating the bindings.

If you build a package that is used as a dependency by your projects that artefact may be archived, but that is different from committing generated files to source control.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.