We're using a similar concept to Event Sourcing to store the data/actions of a collaborative system we have running.

The system has some 2-3k users that use it daily and it's growing.

As time goes by, the codebase evolves and the data we already have stored needs to be updated to match newer formats. This is a rare scenario but it happens and I believe it's far "healthier" to convert past data to conform to the new codebase, rather than handling previous formats within the codebase thus littering it as time goes by.

I've written a small tool that downloads chunks of events (~3000 events) per cycle, map()s over them thus converting them and batch-updates it back. It's quite efficient.

However, it just took me almost a day to convert 7 months worth of data (~30GB). After a year or so this might start to take 2 days to finish and after 2 years this might start to take weeks.

Are there better strategies when handling such data conversions?

  • What Event store implementation are you using? Feb 21 '18 at 9:52
  • @ConstantinGalbenu Not sure - It's just a table that inserts an event as a row in it. Nothing too fancy. Feb 21 '18 at 9:54
  • So, it your own implementation. Feb 21 '18 at 9:54
  • And how you do it, you create+build a separate table and then swap them when the conversion is done? Feb 21 '18 at 9:56
  • @ConstantinGalbenu That's correct Feb 21 '18 at 9:57

For the handling of such a system over years, you need to look for overall strategies which work on long terms, not for some local optimizations. But the very first thing you should to ask yourself here is:

Do I really need an optimization right now?

Consider that in two or more years, many things can happen. Maybe the hardware or the underlying DB software gets faster, which could solve your problem automatically. Or you can scale up by using more servers, since renting them gets cheaper. Or your system will vanish from the market, since users change to some competitor. Or you changed the whole underlying architecture in a newer version of the system, so the solutions you worked out now won't be applicable any more.

Be careful not to invest too much effort for things which may never apply.

But let us assume for a moment your system will be online for the next 10 years and constantly growing over that time. I would focus on two things here:

  • archiving older data when it is not used any more (so you can remove it from the live database)

  • avoid the need of frequent data conversion by implementing enough flexibility into the system it can handle the most frequent requirement changes without database schema changes, or by backward compatible database changes.

If, in some years, you really come to the point where you need to convert a Terabyte of data to a new data model, you can implement some strategy where you

  • migrate all existing events at date "x" from the live system to the new data model (lets says this takes a week)

  • then migrate the events from "date x" to "date x+7" from the live system

  • then switch of the system for a few minutes, migrate the remaining events, and

  • finally switch over to the new data model.

However, if I were in your shoes, I would not implement this before it turns out it is really required.

  • The strategy you point out in the 2nd paragraph is exactly what I'm going for right now. Point taken though - perhaps it's better to examine whether I could somehow translate the events to newer versions at run-time rather than converting persisted data over and over again. Feb 21 '18 at 10:53
  • @NicholasKyriakides: how often do you need to apply such a data migration currently? Once a week? Once a month? Once a year?
    – Doc Brown
    Feb 21 '18 at 10:56
  • I'd say once every 4-5 months. I might sound a bit pedantic since it happens so rarely, but long conversion timespans carry with them a lot of secondary problems (slow release cycles, problems during conversion are costly to fix since I might need to re-run the conversion etc..) Feb 21 '18 at 10:56
  • @NicholasKyriakides: and is it really a problem when this takes some days? I mean, because of the nature of the event store, if I got this right, you don't need to take the system offline (except for the short period where you have to switch over to the new model).
    – Doc Brown
    Feb 21 '18 at 11:00
  • 2
    @NicholasKyriakides: the problem of slow release cycles can be mitigated by making only backward compatible schema chages for minor updates, and keep the non-backward compatible changes to major updates.
    – Doc Brown
    Feb 21 '18 at 11:04

The tactics/optimization that you could use to upgrade an event to another event in the Event store is this:

  • copy the entire events table to a temporary table (CREATE TABLE NewEventStore LIKE EventStore - to preserve indexes)
  • select the event types that have changed using a WHERE eventName IN ('event-type1', 'event-type2'); you should have an index on eventName for fast select
  • modify/replace each of the selected event with the new version;
  • stop the writes&reads to the EventStore
  • swap the old table with the new table (DROP TABLE EventStore; RENAME TABLE NewEventStore to EventStore)
  • re-enable the writes&reads to the EventStore

If you do want to keep the system live during the swapping then things get more complicated as you need to synchronize somehow to the writing system.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.