I was reading the Continuous Integration by Martin Fowler article yesterday, the article is from 2006 but I think it is still very applicable today, the statement around a single command stuck in my head.

Automated environments for builds are a common feature of systems. The Unix world has had make for decades, the Java community developed Ant, the .NET community has had Nant and now has MSBuild. Make sure you can build and launch your system using these scripts using a single command.

A common mistake is not to include everything in the automated build. The build should include getting the database schema out of the repository and firing it up in the execution environment. I'll elaborate my earlier rule of thumb: anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine.

The concept sounds great but with my limited working knowledge of Azure Pipelines for Continuous Integration builds it feels counter-intuitive to not use the tasks available in Azure Pipelines and have this run a single script that we wrote.

On the flip-side it doesn't make sense to just have a build definition in Azure Pipelines as being able to use the same build definition locally during development is useful.

I would guess the same applies to other applications offering CI functionality where the build pipelines do not support being reused locally. I read that GitHub Actions can be executed locally, perhaps that is the exception?

How is continuous integration done in the real world to allow builds to be done locally and on the CI server?

For context I am working with Microsoft solutions/projects i.e. .csproj, .sqlproj

  • What you describe is a common dilema I see when build automation / continuous integration is considered. While it is possible to create such a script, it often means giving up lots of CI-only tools. My experience is that most people don't consider this and just see CI as the source of truth, with local development being mostly manual setups. – Euphoric Aug 13 '20 at 9:11

I think everyone strives to accomplish the Fowler's direction, even if in the real world this doesn't always happen. the paramount is the Reproducibility, but tools alone won't help. We need to set up procedures and practices to obatin it. In particular, to me, there are 2 important principles:

Declare What to Do

The problem here is not how many steps to perform, but where the pipeline configuration is declared and how is maintained. For instance, Jenkins let to define pipelines through groovy scripts that are versioned and maintained from developers. In some extent, Jenkins is "stateless". The only contract between the server and the devops-defined pipeline is the avaliability of the plugins used from the pipeline....but the plugin confguration, as well as other specific settings, are stored into xml files that could be edited, dumped ad restore as soon as needed. So applying these practices grants you that should be able to destroy your Jenkins cluster, recreate it and start building. So to reproduce the build configuration we need to set up things in a way that everything is declarable, storable and appliable

Declare How to Do It

Aside from specific tools used to implement the CI concept, CI servers should behave like "artificial developers": they should do same things as devs do to test the application locally, and just add a further step: publish the artifact to a repository to make it ready for continuos delivery

Usually i like to create a Makefile with few targets:

  • run infrastructure (local DB, Local MQ, some fake external service)
  • build application
  • build application + run application
  • run infrastructure + run unit/integration tests

using some tools to automate the depencencies deployment (namely, docker and docker compose) I'm able to pick a brand new laptop with some tool installed and start working. What does prevent CI servers to run same commands? The only difference between CI server and us

  • their ability to perform concurrent build (so also multiple DB instances running)
  • the fact that every build step should never have an intermediate manual input, that is easy to accomplish if we choose to feed the project with config files instead of prompts
  • they uses different credentials or different auth methods to interact with external services (package repositories or even kubernetes)

So abstract away build steps using script won't just help CI, but also poor devs that are scared to build new features because of the endless time spent setting the environment and stuff

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.