I'm working with a NoSQL DB with eventual consistency. My software is not only inserting Java Objects in JSON but also creating secondary indices for cross-referencing and similar things. Due to the fact that my software has to create all references on it's own, there is no real mechanism to ensure complete consistency throughout the whole DB.
That's why I want to program something like a housekeeping routine, that can be called during runtime to test for inconsistencies.
Now I'm wondering if there are any advisable software patterns that serve these specific purposes:

  • A scheme (not quite sure what the format will be; probably XML or sth similar) should be used as input that describes the relations between DB entries. This scheme will probably be the documentation itself.
  • The routine should be able to identify keys that were never accessed during checking the relations mentioned above.
  • It must be capable of checking aggregated values, de-/incrementation and JSON.

The first item on the list is the most important, so that the routine identifies flaws in the code as well as in the documentation/requirements. Any hints towards helpful literature or probable issues, I should consider, are much appreciated.

  • 5
    I fear there is already a tool for managing relations between database entries and maintaining their consistency; it's called a relational database. – user253751 Sep 16 '15 at 2:43
  • @immibis true enough : ) But in this case, the in-memory database is not meant to be replaced. So your comment is not helpful and not even the slightest addressing my question if there are software patterns that help realize the listed points above ^^ it doesn't matter if they solve these problems partially or are mostly used with topics not concerning databases. – Unknown Id Sep 16 '15 at 7:56
  • 3
    Do realize that 'NoSQL DB' is not specific enough. There are some NoSQL databases that have eventual consistency, and others that have immediate, and some that have none. You need to specify exactly which NoSQL database you are using as the answer for a KVP is going to be different than a graph or object database. – user40980 Sep 21 '15 at 0:19
  • @Michael : good to know. A redis database is used, so KVPs. – Unknown Id Sep 21 '15 at 7:42

First of all, eventual consistency should be pretty safe, and such a "data housekeeping" process may not be mandatory. It may be indeed a feature to have several states of the data in the database. In BigData process, it is pretty usual - Google for instance may have several states of the same data in its servers.

IMHO your question may have answers at two levels, following the DDD architecture.

1. At Infrastructure Layer

You may write some code specific to the NoSQL engine you are using.

It would be optimized e.g. for fast process using KVP, perhaps with the following patterns:

  • Using memory as cache as much as possible, to avoid unnecessary requests on the NoSQL;
  • Run the operation in a separated process, to reduce impact on the main process;
  • Let this separated process use a map/reduce pattern, then injecting the modifications as batches;
  • Use some kind of CQRS pattern, and several databases: one KVP database for your direct writes, then dedicated secondary read-only databases, containing some "polished" data (also filled with map/reduce), ready to be consumed by your Domain Services.

Your idea of identifying "dead" keys is interesting. It could be implemented within the infrastructure layer, as part of the storage process, e.g. by maintaining an in-memory list of just written entries, then deleting then garbage-collect them using a generational-like algorithm, marking keys with a generation number, ready to remove the unused one.

2. At Domain Level

In DDD we try to be as uncoupled from the database as possible. As a result, doing some data cleaning process at infrastructure layer sounds somewhat weird.

Such data cleaning may be part of the domain, in a dedicated Bounded Context. The infrastructure layer, when apply the map/reduce algorithm, may use some logic defined in the domain, using domain objects, to validate your data, and fix any synchronization problem.

Last but not least, if your data seems eventually incoherent, it may smell like a domain problem. Your aggregates may be wrongly defined. What you should persist are Aggregates Roots, which scope has been defined by a given execution context. Typically, Aggregates are persisted in your database, and guarantee the consistency of changes by isolating their members from external objects (i.e. you can link to an aggregate via its ID, but you can not directly access to its internal objects). The eventual consistency of the KVP should be enough to ensure consistent data. If you encounter consistency issues, you may consider fixing your domain, which may have some problems about the Aggregate Roots scope.

|improve this answer|||||
  • Thanks for that great overview! In my case, CQRS combined with garbage collection turned out to be a great combination to analyze my database : ) – Unknown Id Oct 14 '15 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.