We're using a similar concept to Event Sourcing to store the data/actions of a collaborative system we have running.
The system has some 2-3k users that use it daily and it's growing.
As time goes by, the codebase evolves and the data we already have stored needs to be updated to match newer formats. This is a rare scenario but it happens and I believe it's far "healthier" to convert past data to conform to the new codebase, rather than handling previous formats within the codebase thus littering it as time goes by.
I've written a small tool that downloads chunks of events (~3000 events) per cycle,
map()s over them thus converting them and batch-updates it back. It's quite efficient.
However, it just took me almost a day to convert 7 months worth of data (~30GB). After a year or so this might start to take 2 days to finish and after 2 years this might start to take weeks.
Are there better strategies when handling such data conversions?