How to manage concurrent build & test toolchains in (embedded) software development?

Question

TL;DR: How do you manage the complex development toolchain in different (embedded) projects?

In our company we are discussing the toolchain setups in different development teams and how to work together in the future. It is important, that for each project the setup is fixed (e.g. specific versions of all tools like python, compiler etc.) while we need to switch between project frequently (sometimes working in multiple projects on the same day). So we need a unified way how to connect a project to the correct toolchain without spending time on configuration each time we switch. So far we have identified 3 different approaches:

1. Local Setup: All devs have the toolchains in all necessary versions installed locally on their dev machines. Projects all have scripts that specify the toolchain with local paths.
2. Shared Setup: The tools are installed/hosted on some common accessible server or network share. Projects all have scripts that specify the toolchain with shared paths.
3. Containerized Setup: The projects contain configurations for containers which host the tools, e.g. with docker.

There are many advantages and disadvantages:

• ad 1): All machines have to be the same setup to be able to use the same scripts. But it is easy for developers to "play around" and experiment when needed before committing when defining/changing the toolchain. Breaking something only has impacts on your own machine. It is also easy to work on different projects when you have the toolchains all available and can start one IDE instance per project. Tools for legacy projects can be reinstalled when needed and local configuration adapted if necessary.
• ad 2): The tools are centrally managed which makes updates cumbersome. Frequently there are problems with missing permissions. Tools need to be supported for a long time for legacy projects (years and years), and once a dependency is there, it is not allowed to change something. It is easy to onboard new people since everything is ready available.
• ad 3): (Almost) everything needed to work in a project is included in the repository, dependencies on local or shared setup are minimal which promotes exchange and collaboration. No risk of external meddling with the toolchain. But setting up the toolchain is more work and having different tools possibly in different containers interact needs even more effort. Especially in embedded development it is problematic to have e.g. a debugging tool interact with hardware when run in a container. Also it is questionable how this will impact RAM and efficiency when working in multiple projects in parallel.

It is of utmost importance, that a toolchain for a project can be recreated even after years when bugfixes in a certified & delivered software are needed.

Is there any informational material or experience that you can share?

Answer 1

We also have the requirement that we need to support our embedded software for 15 years. We have created a solution that looks a bit like the containerized option you describe using Conda environments.

We have a custom bootstrap script that checks if Conda is installed on the machine and if not installs it. After that, a virtual environment is created with the tools that the project lists as dependencies. The dependency lists are versioned (by creating a hash), so that we can re-create the build environment as it existed for a particular commit.

The only tools that do not get installed automatically are tools that require a software license, as license management is not included in our setup. For us, this is primarily relevant for the compilers. Those need to be installed separately (once, outside the virtual environments) and we have a package in our Conda setup that can check if the correct compiler is available on the system.