Common libraries in build chain

Question

Like many (most ?) people, we have multiple "common" libraries managing various things (business objects, utilities, external libraries...) for multiple projects (web service, admin/users site, desktop apps...).

As it's set up today, each project copies its binaries in an "assemblies" folder one level higher than the project, and they all reference binaries in this folder, rather than the project itself.
The SVN folders are a mess too, with common libraries in specific projects solutions, and services sometimes grouped, sometimes within their own folder.

Today, there is an incredibly complicated set of Jenkins jobs copying and using various projects outputs all over the place, with dependecies very hard to trace and often breaking : for instance, when I commit to the admin UI, it builds quickly, but fails because the web service hasn't finished building.

I see many problems with this organization ; among others :

• I have to respect a specific folder tree to be able to build a project, I can't just check out a single project and work on it.
• In order to build project A, I have to build project B, and before that project C, and so on and so forth. I can spend a lot of time finding out which project I'm missing to build the first one.
• The whole build configuration is stored in Jenkins, and developers do not have a direct access to it. So I cannot simply checkout a project I haven't touched before and build the whole dependency tree by running a script

I'm used to write large scripts that do everything, so It's driving me mad, but other developers are used to it, and just shrug their shoulders.

How do you configure your build system to handle your common and external libraries ?

(Update : we're using .Net, because apparently there are different solutions for different languages)

---

About dependency managers : nuget, npm, etc

We have thought about using an internal nuget server to store versioned DLLs, and use it to reference other projects.
The problem with this approach is the impact on the daily workflow.

Let's say I store my binaries in a nuget repository. And in order to make a new feature, I have to edit projects A, B and C. My workflow becomes a lot of waiting:

• I edit and commit project A.
• I wait for the A build to finish, and for the nuget feed to update
• I update my nuget package on B, edit, commit.
• I wait for the B build to finish, and for the nuget feed to update
• I update my nuget package on C, edit, commit.
• For my next feature, I must go back to step 1.
• Or, I code multiple features at once, and it's not very agile (and very risky).

This organization completely breaks the coding flow, and as such, is inacceptable.

Maybe I'm not thinking of the proper organization using a dependency manager? Maybe there's a better way?

Answer 1

Usually, API of nuget packages shouldn't change that often. E.g. if the API of package A didn't changed, it is not required to rebuild package B and C because it can be used with the new version of package A.

However, if the API of a package is still under heavy development, an alternative is to include the project's source code as external project (externals for SVN or submodules for GIT) directly in multiple projects. That solution has also some drawback (e.g. you may accidentally break other projects by changing the external projects) but it definitively simplifies the build process ;).

Answer 2

I'm at a .NET shop that has tons of class libraries which we like to reuse across multiple projects and here's how we do it.

• Projects do not live underneath Solutions in the folder structure of our source control (TFS). We have an 'Sln' folders for solutions, a 'Lib' folder for class libraries, 'Web' for webapps, etc.

• If a solution I'm in needs to use an existing class lib, I pull down that library from source then use the "Add existing project" on my solution to select the library.

• We utilize TFS for builds.

• Any time I'm changing a method in a shared library, its my responsibility to search through our codebase to look for other apps that use that library and any methods that I'm changing and to coordinate with other team members on if those other apps need to be re-tested/deployed. (Fortunately not super common)

This flow lets us re-use class libraries, with the libs being fully de-buggable when running an app in Visual Studio, and this process replaced an older process where we compiled the common DLLs to a common location and targeted our custom proj's to grab them from there. The problem with that approach was that it was very hard to tell what version of a DLL had been dropped into the shared location, and they were kind of blind boxes of functionality. I think it also created a case where Test-vs-Prod DLL versions could get mixed up too.

The way we work now is not always ideal (makes dealing with Nuget a pain), but I personally prefer it to just grabbing the DLLs off a shared drive.

Answer 3

External code used to build the application is added using a dependency manager. Maven for Java, Composer for PHP and node/npm for JavaScript. The same mechanism is used for code we wrote internally and code from external repositories. For our own code we use an internal repository which can be as simple as a set of folders on a central server.

If the build includes more then only retrieving the external code a build script is added. Most of the times these scripts can be configured in the build tool (Maven, Composer, node) or at least called from them.

The names of these scripts are the same across projects, called build or test to run the tests. Using the same name makes it easier to start with a project you haven't worked on before.