How to deploy pipeline rerun?

Question

I have an application that's mostly one large data pipeline. That pipeline runs daily and stores processed data in the database (it takes the execution date as its argument). Occasionally the client finds some bug in the data processing algorithm or just simply requests a change in it. Then I write the bug fix or modify the algorithm, push my changes to VCS and I can nicely and easily deploy my changes to the production environment. However this doesn't solve the problem of applying those changes retroactively. There's still a lot of data in the database from the past, processed with erroneous or outdated version of pipeline.

Moreover, I don't want to rerun a whole pipeline. If my pipeline looks like this:

A -> B -> C -> D -> E

and I've made a change only in the second arrow, then I want to rerun only second, third and forth (consecutive arrows depend on the previous ones).

I've been solving this issue manually, namely by deleting old data from some tables in the database and forcing the pipeline to run sequentially with old dates as arguments starting from the proper arrow. Sometimes deletion part requires something more sophisticated like deleting values only from one column - the exact action depends on the algorithm modification.

This creates a new problem - new pipeline runs require old ones to be finished before they can start, otherwise the results would be wrong. So each time I do this I have to mess with production environment even more, disable pipeline autoruns and wait for the retroactive runs to end until I reenable it.

This whole process seems really wrong and error prone. Can I do better? I've been thinking about putting all those actions into small transition scripts each time I need them but then what? How to deploy those scripts to production? It seems really weird to put such one-time use scripts to VCS. They are not a part of application, they're just some ways of transition between versions - the application could be built from the scratch without their existence. On the other hand it resembles a bit database migrations...

But ok, even if did so, this complicates my CD. My runners would need to check what kind of changes were made in the last app version and if they're the algorithm changes they would need to additionally run transition scripts. It seems like a lot of additional and non generic effort to generic looking problem.

Answer 1

So if you start with this:

A₁ -> B₁ -> C₁ -> D₁ -> E₁

And switch to this

A₁ -> B₂ -> C₁ -> D₁ -> E₁

The only useful data you can work with has only been through this:

A₁

Record, with your data, what it's been through and you know what it can be used with.

For example, if you now need:

A₁ -> B₂ -> C₃ -> D₁ -> E₁

You can use data that has only been through

A₁ -> B₂

and

A₁

Answer 2

I see two different problems described in this question:

1. How to implement fixing of the data in case of a bug more smoothly?

2. How to bring such data fixing into production without interfering with the regular processing?

For 1, there is probably no silver bullet. Bugs are nothing which can be easily foreseen, and each correction of existing data always needs an individual solution, tailored to the specific case, there is no way around it. Even if there is already a correction in place for the "usual pipeline processing", this correction does not know how the "wrong" data looked beforehand and how it can be transferred into a "corrected state", that's why you usually cannot reuse the correction "retroactively".

For 2, I would change the design of the system in a way that allows more easily planned maintenance processing. That should be a regular one-time action in production which can be activated at certain points in time whenever its required, outside of the regular pipeline processing.

So in case you need a "retroactive fix", develop it, test it in your dev & test environments, and then deploy the fix to production for this maintenance run. If you do this right, you will not have to mess around with deactivation or activation of the "regular processing", the system will automatically handle this.

Let me add one final recommendation, maybe you already know this, but let's make sure: it is a good idea to make changes to the data - and specificially such data fixing - idempotent and restartable. So when one runs them accidentally twice, no data will be messed up, and when the process was stopped half-way through, just starting it again should process the unprocessed data. That helps a lot to keep things manageble.