1

When shipping software that uses a database, it is common to use a solution to track database changesets.

For instance: you add a feature to manage books, you also have a new SQL script to create that book table. Solutions like Liquibase and Flyway offer frameworks for writing database changesets and they keep track of which SQL scripts have already been executed in a given database. They offer patching automatically (during the next application startup) and on demand.

I am looking for a similar solution to execute scripts, or reasons why I should do otherwise.

Use cases :

Initial data population

You add a movie review feature to your app, and you need to feed it a movies list.

That movie list does not fit in a SQL file and is generated by another program.

A script would allow generating the list (or calling an API that does) and feed it to your app via an API, e.g. /api/import-movies

Data migration

Let's say a movie studio sold its brand in a few countries.

You need to rename it to the new owner in those countries. Using a database changeset is hard because there is some business logic involved.

Pre-deployment check

You would like to deploy a new version only when certain conditions are met (circuit breaker).

Alternatives considered

Using database changesets

Drawbacks:

  • the files can be large and get accidentally corrupted (truncated, re-encoded)
  • they may exceed some upload limits
  • the data may not be static and needs to be dynamically generated
  • they are low-level and bypass business logic, logging, etc.

Modifying the deployment pipeline

It is possible to modify a deployment pipeline to execute scripts before and after a deployment. However this seems impractical when this is often needed.

Also, because of the single-use nature of these scripts, it requires reverting the deployment pipeline right after the deployment was done.

Ideally, we would put scripts in a changesets directory of the software repository (just like it's done with database changesets) and they would be picked up by some CI/CD tool.

7
  • I like this question, I've been wishing there were standard tools for this for years. I imagine they would be very similar to db versioning tools, one difference would be that they'd run each new command after deploying code, not before.
    – bdsl
    Jun 15 at 19:42
  • 3
    While I think this is an interesting question, it is too broad for this site, and borders on requesting tool recommendations (which is off-topic) or requesting assistance with writing code (which is also off-topic). There is no single canonical answer, because it depends on the tools you have available. Jun 15 at 20:38
  • @GregBurghardt I agree. Would it help if I reframed it like "is there prior work about this"? I come from StackOverflow, this is my first question here.
    – pyb
    Jun 15 at 21:16
  • Some of what you need might be aided by configuration management (e.g. chef, ansible, puppet, etc.), although it's unlikely that you'll find any tool which meets all of your specific needs (they generally don't go as far as providing complete solutions to specific problems). Such tools are typically focused on providing frameworks and patterns, leaving developers to define their own apply/rollback scripts and configuration files (although the most popular tools might have popular community modules which may help save reinventing a few wheels). Jun 16 at 4:53
  • 1
    @BenCottrell I got your suggestion now. I love it! It makes sense to describe the desired state of the service in a configuration management, using a deployment pipeline is brittler (pipelines can be recreated, most developers do not have one locally, etc.).
    – pyb
    Jun 16 at 14:40

0

Browse other questions tagged or ask your own question.