Setting up tests and a continuous integration pipeline in a polyrepo environment

Question

In the past I've mainly worked with monorepos, specifically a Vue.js app that was served by a rails backend. Setting up E2E/integration tests in the CI was simple, because all the required parts were kept in one place anyway and the testing tools provided by rails did most of the heavy lifting. Now I'm working on a polyrepo project. The main components are a CMS, a backend and a frontend.

I'm struggling to find a satisfactory setup and would like to discuss the topic in general. I want to have the following things:

• when I change the backend, I want to make sure that it still works together with the frontend
• vice versa, when I make changes to the frontend, I want to make sure that it's compatible with the backend
• when changing both, the changes should be tested against one another

In the monorepo scenario, I would have just spun up the backend and ran tests against it from the frontend. But now those components live in separate repositories. I could pull the code from one into the other, but where should that happen?

• in the frontend: then changing the backend wouldn't trigger E2E tests
• in the backend: same thing
• in a separate repo: frontend and backend would have to push their changes to the E2E-test repo where the tests would run. When both repos change, they just have to push to the same branch name and changes in both repos will be tested against one another. I like the idea, but how do we get feedback in the frontend/backend repos without manually checking the E2E-repo? Also, the setup seems quite complicated.

In theory, the components should probably be completely isolated from one another and be able to evolve independently. In practice, this would imply a lot of overhead, especially API versioning and backwards compatibility.

One options would be to move all components into a monorepo. This might be the best choice, but I only have three repositories, so this should be a "simple" problem, compared to a microservice architecture and I'd like to hear the alternatives. How would one approach this in general? Is my mindset all wrong and the E2E tests should better run on a staging or production server while only the unit and API tests run within the CI? There should also be a simple solution to run the E2E suite locally, shouldn't there?

I know that there is no clear-cut answer and that any answer will probably be opinionated but I'd like to gather ideas and approaches to the problem to figure out a good solution.

Answer 1

E2E Tests should depend upon built, versioned artefacts, not branches or source control.

In the same spirit of manual QA testers being reliant upon built, versioned artefacts, automated E2E tests should ideally follow the same mantra. For one, it's valuable to be able to redeploy those exact same artefacts in another environment if there's a need to reproduce any problems found (avoiding any possible variability which may be caused by rebuilding).

Problems may not necessarily always be in the code but could be down to the CI process itself that produced the artefact (or even in the deployment and configuration) so it is often more useful for E2E tests to identify problems with a specific version rather than a branch. Similarly, being able to redeploy the same artefacts elsewhere could more easily narrow down specific environmental issues, or even problems in the deployment process.

Versioned Artefacts

When the CI process for a deployable software component is completed, I would typically expect one of the final steps to be for the CI server to bundle the artefact into some kind of archive or package (For example, a version-numbered folder or version-numbered zip), and for it to publish that archive/package into some shared space such as a package repository or file server, ready for deployment.

E2E in CI

If the E2E tests have their own separate CI process, a pre-requisite would often be the need to specify linked artefact versions that those tests must be run against (for example, in a configuration file as part of the test source which the CI process will read, then it can trigger deployment for the specified versioned artefacts). This eliminates any potential ambiguity about what the tests had been run against, and indeed the source for the E2E tests would contain those artefact version numbers so that it's clear which versions the tests are intended to target.

One of the disadvantages of linking tests to a branch or source repository is that the 'Head revision' at the point in time when a test fails may be different to the head revision when a developer moves to investigate, so traceability is lost.

Again, in line with how this would work for a manual QA tester, part of the test reporting would always specify the exact versions of each artefact that those tests had been run against, so that someone trying to investigate a problem would be able to look at the correct code revision and re-create the environment more easily.

Local Testing

There will always be differences between running E2E tests locally versus running them in the CI test environment; not least because developers' own local environments are different, but also a developer is likely to have checked out, built and may want those projects running inside an IDE with a debugger attached, so they would probably not want a pre-built artefact.

A developer also typically would not need or want to run through a versioning and deployment process for source code which they're building and running locally, so the E2E tests would most likely need to run against a different configuration altogether.

Versioning

Typically it's desirable to trigger E2E tests against "feature branch" releases in CI before any code reviews or merges have taken place;

I would suggest ensuring that all builds in CI are able to produce artefacts, even those which are "unstable" - it may be useful to ensure that the CI process distinguishes pre-release artefact versions from those branches, for example: 1.23.456.789-unstable or 1.23.456.789-prerelease.