7

I am facing problems structuring projects and libraries.

In the company I am working for I often see, that things would be more maintainable and less error prone, if we could extract common code and build libraries with that code.

A simple example may be the usage of a custom logging library instead of copying code from other projects or, even worse, writing everything again from scratch.

So wanting to expose things into libraries there comes up the question how to do that? My idea is to put every lib(or related set of libraries) into its own repository. But this would mean, that to build the project, you definetly need to check out at least two repositories. Because I have been workong on a project with dependencies to a lot of other repositories I became quite careful with creating dependencies to other projects.

So my question is: is there another nice solution for this? What do you guys do?


There are more than 150 repositories, which are not all related of course. Most of the projects have their own repository. To provide a scenario, lets assume the following:

  • Application A
  • Application B (already references 30 other in-house repositories)
  • LibXYZ

Both applications need to use LibXYZ.

Versioning:

Versioning depends on the project. The older projects just took the revision number from SVN. We are now moving to versioning with fix numbers like 1.2 or 1.2.5, so let's assume this as versioning approach. Every release is tagged in SVN.

Plattform:

We are using Qt targetting mainly Windows. But I appreaciate a cross-plattform approach very much.

What prevents me from just creating a repository for common code?

The problem I see is that it adds a lot of complexity to a project. To be able to compile and run an application I need to:

  • check out multiple repositories
  • compile all depencies
  • copy all compiled DLL's into the application's build directory

All this again has to be documented and maybe also to be included in the build scrips. Moreover this forces the other team members to have the exact same setup as I do.

I am not saying that this is impossible, I am just searching for a better approach.

  • Sorry, but I'm not seeing the problem here. You're going to need at least one more repository containing the common code. It can potentially be just one. What prevents you from doing that? – Robert Harvey Jan 9 '17 at 14:30
  • @RobertHarvey I edited my question to answer your question, too. – exilit Jan 9 '17 at 14:45
  • 3
    It seems like what you want is a repository manager. I'm not familiar with Windows development, so can't give you advice as to what repository manager you might want (that would probably be considered off-topic anyway). But be aware that you will need to change your build scripts. – kdgregory Jan 9 '17 at 15:12
  • 1
    @RobertHarvey - because I'm not familiar with Windows or C++ development, so I would be writing about generalities rather than giving specific advice. Although I suspect the answer involves NuGet. – kdgregory Jan 9 '17 at 17:55
  • 1
    @kdgregory: We actually tried NuGet as a way to manage our internal repositories. It caused more problems than it solved. Instead, we just maintain a shared folder with the common assemblies. It's probably not the best way to do it, but it has the virtue of simplicity. – Robert Harvey Jan 9 '17 at 17:58
4

To adress these problems one by one:

check out multiple repositories

Using SVN, you can utilize the "externals" feature for this. This will allow you to define dependencies between different repos in a way where you either tell "app A depends on lib B tagged with version X", or "app A depends on lib B, always the newest version". For other source code control systems without a feature like "externals" (or if "externals" does not provide the solution you are looking for), one can implement a similar behaviour by writing some checkout scripts.

compile all dependencies

Your build system should handle this for you, once you defined the dependencies there properly. There is no difference to the situation when you use 3rd party libs. You can reduce compile times by providing precompiled binaries of each library release and put the compiled binaries in into the repo, too.

copy all compiled DLL's into the application's build directory

Yep, this should happen automatically by the step before.

In general, the key to this is automation. Whenever you find a recurring, manual task which might become error-prone (like checking out from the "correct" repos, copy file X to folder Y, set a tag with increased version number here, and so on),

  • write the manual steps down into something like a "readme" file
  • figure out how to automate those steps through scripting
  • make sure the scripts do proper logging and error handling! Checking preconditions and some failure tolerance is also helpful in my experience.

When dealing with multiple applications, repos and libraries, it might be also a good idea to set up a build server which checks out everything nightly and does a full build of everything.

Moreover this forces the other team members to have the exact same setup as I do.

To some degree, yes, but this is not necessarily a bad thing. For each development tool, I recommend to add a readme file describing the specific setup requirements in your inhouse environment. Your build scripts should rely as much on relative paths as possible, do not use any absolute path when not needed. And if necessary, have some central initialization script figuring out things like the installation paths for the required tools on the current machine, and set up some environment variables accordingly.

  • Even though here is nothing really new to me, I'd like to thank you for the nice write up summarizing the most important points. If there won't be other answers, the conclusion for me is that my initial thoughts have been correct and currently there is (most likely) no better solution. – exilit Jan 10 '17 at 8:17
  • 1
    @exilit: the key is to combine these measures and to look for simple solutions. Convention over configuration helps a lot. Establish conventions how you name your build/checkout/tag/deploy scripts, how to format tags, how to structure your repos, how to structure your build directories, how and where to document things. Then the complexity stays controllable. – Doc Brown Jan 10 '17 at 9:26
  • Putting compiled binaries into the source code repository frightens me. Why not have an FTP server or similar infrastructure provide them? There are also CI servers that can automate this process without any manual scripting. Otherwise, I like this answer a lot. – 5gon12eder Jan 10 '17 at 13:11
  • "Putting compiled binaries into the source code repository frightens me." - this is an old standard technique for this use case, nothing invented by myself. – Doc Brown Jan 10 '17 at 13:51
  • .. The advantage is, you can include the binaries under the same tag as the source code. And yes, if one has a running CI server at, it is perfectly possible to grab the binaries from there instead, there is surely more than one way to realize this kind of caching. – Doc Brown Jan 10 '17 at 13:57
2

Don't reference other repositories. You have no requirement to build the dependencies every time you build the main application.

Build each library repository into a dll separately and version that dll

add the compiled binary dll to a package repository (nuget npm)

add a reference to the package on the package repository to the main project

when you build the main project, the first step is to download the versioned dlls

(if you haven't got a package manager, add the dll to source control on your main repository)

  • Your supposed alternative solution is fine. Only your first paragraph misses somewhat the point, since referencing a repo with a binary DLL + the header files of a C++ library as "externals", it can solve the OPs problem without forcing a new build of the original lib. I think with a few edits, by not wrongly pretending this is the "only correct" solution, this could become a good answer. – Doc Brown Jan 11 '17 at 6:34
  • 1
    I agree that the main point is to reference the binaries rather than import the source. where you store the binaries is secondary. But supposing the lack of a package manager, I would put them in the main repo, rather than the library repo for various reasons – Ewan Jan 11 '17 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.