When shipping software that uses a database, it is common to use a solution to track database changesets.
For instance: you add a feature to manage books, you also have a new SQL script to create that book table. Solutions like Liquibase and Flyway offer frameworks for writing database changesets and they keep track of which SQL scripts have already been executed in a given database. They offer patching automatically (during the next application startup) and on demand.
I am looking for a similar solution to execute scripts, or reasons why I should do otherwise.
Use cases :
Initial data population
You add a movie review feature to your app, and you need to feed it a movies list.
That movie list does not fit in a SQL file and is generated by another program.
A script would allow generating the list (or calling an API that does) and feed it to your app via an API, e.g.
Let's say a movie studio sold its brand in a few countries.
You need to rename it to the new owner in those countries. Using a database changeset is hard because there is some business logic involved.
You would like to deploy a new version only when certain conditions are met (circuit breaker).
Using database changesets
- the files can be large and get accidentally corrupted (truncated, re-encoded)
- they may exceed some upload limits
- the data may not be static and needs to be dynamically generated
- they are low-level and bypass business logic, logging, etc.
Modifying the deployment pipeline
It is possible to modify a deployment pipeline to execute scripts before and after a deployment. However this seems impractical when this is often needed.
Also, because of the single-use nature of these scripts, it requires reverting the deployment pipeline right after the deployment was done.
Ideally, we would put scripts in a changesets directory of the software repository (just like it's done with database changesets) and they would be picked up by some CI/CD tool.