How do we avoid CI-driven development...?

Question

I'm working on a very large research-led open-source project, with a bunch of other regular contributors. Because the project is now quite big, a consortium (composed of two full-time employees and few members) is in charge of maintaining the project, the continuous integration (CI), etc. They just don't have time for integration of external contributions though.

The project is composed of a "core" framework, of about half-a-milion-or-so lines of code, a bunch of "plugins" that are maintained by the consortium, and several external plugins, most of which we aren't even aware of.

Currently, our CI builds the core, and the maintained plugins.

One of the big issue we face is that most contributors (and especially the occasional ones) aren't building 90% of the maintained plugins, so when they propose refactoring changes in the core (which these days happens on a quite regular basis), they checked that the code compiles on their machine before making a pull request on GitHub.

The code works, they're happy, and then the CI finishes building and the problems start: compilation failed in a consortium-maintained plugin, that the contributor did not build on his/her machine.

That plugin might have dependencies on third-party libraries, such as CUDA for instance, and the user does not want, does not know how to, or simply can't for hardware reasons, compile that broken plugin.

So then - either the PR stays ad aeternam in the limbo of never-to-be-merged PRs - Or the contributor greps the renamed variable in the source of the broken plugin, changes the code, pushes on his/her branch, waits for the CI to finish compiling, usually gets more errors, and reiterates the process until CI is happy - Or one of the two already-overbooked permanents in the consortium gives a hand and tries to fix the PR on their machine.

None of those options are viable, but we just don't know how to do it differently. Have you ever been confronted to a similar situation of your projects? And if so, how did you handle this problem? Is there a solution I'm not seeing here?

Answer 1

CI-driven development is fine! This is a lot better than not running tests and including broken code! However, there are a couple of things to make this easier on everyone involved:

• Set expectations: Have contribution documentation that explains that CI often finds additional issues, and that these will have to be fixed before a merge. Perhaps explain that smallish, local changes are more likely to work well – so splitting a large change into multiple PRs can be sensible.

• Encourage local testing: Make it easy to set up a test environment for your system. A script that verifies that all dependencies have been installed? A Docker container that's ready to go? A virtual machine image? Does your test runner have mechanisms that allows more important tests to be prioritized?

• Explain how to use CI for themselves: Part of the frustration is that this feedback only comes after submitting a PR. If the contributors set up CI for their own repositories, they'll get earlier feedback – and produce less CI notifications for other people.

• Resolve all PRs, either way: If something cannot be merged because it is broken, and if there's no progress towards getting the problems fixed, just close it. These abandoned open PRs just clutter up everything, and any feedback is better than just ignoring the issue. It is possible to phrase this very nicely, and make it clear that of course you'd be happy to merge when the problems are fixed. (see also: The Art of Closing by Jessie Frazelle, Best Practices for Maintainers: Learning to say no)

  Also consider making these abandoned PRs discoverable so that someone else can pick them up. This may even be a good task for new contributors, if the remaining issues are more mechanical and don't need deep familiarity with the system.

For the long-term perspective, that changes seem to break unrelated functionality so often could mean that your current design is a bit problematic. For example, do the plugin interfaces properly encapsulate the internals of your core? C++ makes it easy to accidentally leak implementation details, but also makes it possible to create strong abstractions that are very difficult to misuse. You can't change this over night, but you can shepherd the long-term evolution of the software towards a less fragile architecture.

Answer 2

Building a sustainable plugin model requires that your core framework expose a stable interface that plugins can rely on. The golden rule is that you can introduce new interfaces over time but you can never modify an already published interface. If you follow this rule, you can refactor the implementation of the core framework all you want without fear of accidentally breaking plugins, whether it is a consortium-maintained one or an external one.

From what you described, it sounds like you don't have a well-defined interface, and that makes it difficult to tell if a change will break plugins. Work towards defining this interface and making it explicit in your codebase, so that contributors will know what they should not modify.

Answer 3

To be honest, I don't think you can handle this in a better way - if changes result in breaking maintained parts of your project the CI should fail.

Does your project have a contributing.md or something similar to help new and occasional contributors with preparing their contributions? Do you have a clear list, which plugins are part of the core and need to stay compatible?

If it is hard to build everything on a machine due to dependencies etc you could think about creating ready-to-use docker images as build-environments for your contributors to use.

Answer 4

so when they propose refactoring changes in the core (which these days happens on a quite regular basis), they checked that the code compiles on their machine before making a pull request on github.

So I think this is where the loose style of open source projects can fall down; most centrally-organised projects are wary of core refactoring, especially when it crosses an API boundary. If they do refactor an API boundary, it's usually a "big bang" where all the changes are scheduled at once with an increment to the API major version, and the old API is maintained.

I would propose a rule "all API changes must be planned in advance": if a PR comes in that makes a backward incompatible change to the API, from someone who has not been in contact with the maintainers to agree their approach in advance, it simply gets closed and the submitter pointed at the rule.

You will also need explicit versioning of the plugin API. This allows you to develop v2 while all the v1 plugins continue to build and work.

I would also question a bit more why so many core refactoring and API changes are being made. Are they really necessary or just people imposing their personal taste on the project?

Answer 5

Sounds like the CI process needs to be tighter, more comprehensive and more visible to contributors before they raise a PR. As an example, BitBucket have a pipelines feature that allows this, where you give it a file that defines in code the CI build process, and if it fails, the branch is prevented from being merged.

Regardless of the technology, providing automatic builds when a contributor pushes to a branch will give them much quicker feedback of what gotchas to look out for when making changes and will lead to PRs that don't need fixing up after the fact.

Design issues would be good to fix, but are orthogonal to this problem.

Answer 6

The code works, they're happy, and then the CI finishes building and the problems start: compilation failed in a consortium-maintained plugin, that the contributor did not build on his/her machine.

That plugin might have dependencies on third-party libraries, such as CUDA for instance, and the user does not want, does not know how to, or simply can't for hardware reasons, compile that broken plugin.

Your solution is simple: lower the barrier to contribution.

The simplest way to (1) speed up the edit-compile-test cycle and (2) smooth environment differences is to provide build servers:

• Pick up beefy machines: 24, 48 or 96 cores, 2GB RAM/core, SSD, to speed up compilation.
• Ensure they have the right hardware: FPGA, Graphic Card, whatever is needed.
• Create a Docker image with all the necessary software libraries pre-installed.

And then open those build servers to contributors. They should be able to remotely login in a fresh Docker image, and remotely edit-compile-test on this machine.

Then:

• They have no excuse for not building/testing the maintained plugins: they have everything available.
• They do not have to wait for lengthy feedback with CI-driven PRs: they have incremental compilation, and the ability to debug (rather than guess).

In general, build servers can be shared across multiple contributors, however when special hardware peripherals are involved it can be necessary for a contributor to use said peripheral by themselves.

---

Source: working on software using FPGAs, given the price of the beasts, and the variety of models we need, you don't find each model of FPGA installed on every developer's machine.

Answer 7

If contributing to the core without changing any contract can break dependent software, it suggests that either:

• The contracts of your interfaces may be ambiguous. Maybe adding attributes on your functions and function parameters would help in exposing additional constraints to client code to make the contracts clearer. Or if you’re applying contract-breaking changes, maybe adopting semantic versioning can help.
• The unit tests are not covering enough of the possible call scenarios.

Either problem should be easy to solve, but you mention the core team might not have the capacity to do so. One option would be to ask the community for help in addressing the issue.

Answer 8

No one else seems to have raised this as a potential solution.

• list all plugins that you can access.
• run all the tests that these plugins define
• record all request/responses/interactions between the core and all plugins
• store those recordings, these are now rough compatibility tests.

When developing the core, encourage developers to run these compatibility tests. If they fail do not check in.

This will not 100% ensure compatibility but it will catch a lot more issues and early.

A secondary benefit is that these recordings can highlight which interfaces are actively used, and what features are being actively used.

Answer 9

I'm having trouble understanding the situation as it appears to be: The CI only builds one branch?

Is there a reason you can't build more than one branch with the CI?

The simplest solution to this problem would be to make it possible for any contributor to run the CI build on his/her feature branch.

Then you simply require a successful CI build on the feature branch in order for that branch's pull request to be accepted.