I have an application that's mostly one large data pipeline. That pipeline runs daily and stores processed data in the database (it takes the execution date as its argument). Occasionally the client finds some bug in the data processing algorithm or just simply requests a change in it. Then I write the bug fix or modify the algorithm, push my changes to VCS and I can nicely and easily deploy my changes to the production environment. However this doesn't solve the problem of applying those changes retroactively. There's still a lot of data in the database from the past, processed with erroneous or outdated version of pipeline.
Moreover, I don't want to rerun a whole pipeline. If my pipeline looks like this:
A -> B -> C -> D -> E
and I've made a change only in the second arrow, then I want to rerun only second, third and forth (consecutive arrows depend on the previous ones).
I've been solving this issue manually, namely by deleting old data from some tables in the database and forcing the pipeline to run sequentially with old dates as arguments starting from the proper arrow. Sometimes deletion part requires something more sophisticated like deleting values only from one column - the exact action depends on the algorithm modification.
This creates a new problem - new pipeline runs require old ones to be finished before they can start, otherwise the results would be wrong. So each time I do this I have to mess with production environment even more, disable pipeline autoruns and wait for the retroactive runs to end until I reenable it.
This whole process seems really wrong and error prone. Can I do better? I've been thinking about putting all those actions into small transition scripts each time I need them but then what? How to deploy those scripts to production? It seems really weird to put such one-time use scripts to VCS. They are not a part of application, they're just some ways of transition between versions - the application could be built from the scratch without their existence. On the other hand it resembles a bit database migrations...
But ok, even if did so, this complicates my CD. My runners would need to check what kind of changes were made in the last app version and if they're the algorithm changes they would need to additionally run transition scripts. It seems like a lot of additional and non generic effort to generic looking problem.