0

The team i'm in is moving to use couchbase rather than elasticsearch DB. Now this is only an example but my question is more general. In elasticsearch you have the notion of parent-child relationships . When we move to a standard key value store we don't have these. My Question:

Assuming I do want to enforce relationships such as:

  1. When parent is deleted all children should be deleted.
  2. When child is deleted parents which had a pointer to this child should be updated.
  3. Child is rather large doc so we have a pointer from parent to child.

Which design would yo go with?

Option 1: Build a general layer on top of the KV store which knows to handle parent child relationships such that your programmers only need to specify relationships and your generic sophisticated-dao layer will know to handle these relationships.

Option 2: Relationships should not be managed by DAO layer so the above logical layer should manage it. No generic solution, each domain entity which has this relationships should write their own code in logical layer to manage this layer.

Option 3: Parents hold children in same doc - one doc (both parent and children) - not going to do that, children are too big, and there are other types of children as well. (more reasons I can specify if needed).

Option 4: Use specific db features - I want to avoid that so next time we migrate into a different db I won't need to face this problem yet again. A solution that will work for any KV.

Option 5: Don't move to KV store - move to a relational database, or keep current db(updated by thimothy comment), this is beyond my control, we will move to KV, so this option is out of question. There topmost preference is for speed above any db features.

Option 6: Have an async process which manages the relations, so if I delete a child (in our child-parent example), then if a client gets the parent, then fetches each of child by it's id it would see a child does not exist, only 30 min later an async process will see this inconsistency and fix it, in our case the client would need to decide if this inconsistency is a real error or a temporal inconsistency. This is a possible solution so named as option 6 however I fear that each developer would get who knows where errors and would need to decide whether this is an ok state and to ignore or not.

Option 7: Event sourcing, so that I keep replaying and rerunning at least or compensating in case of a failure where parent is updated and child is not, but I think this does not solve the whole thing child/parent, i would still have multiple commands and events and it's not like one transaction, right now although I would be happy to use event sourcing i'm not at this point maybe in future on a total refactor of codebase, no immediate remedy this can give me.

I know that this kind of questions tend not to have a definite answer I would like to know if anyone faced this general question before, and wether it's best to come up with a general framework a general DAO which knows to manage these relationships or let any developer manage his own relationships.

Please take into consideration that giving each developer the responsibility to manage his own relations is dubious because the more developers you have the more you want to have some base infrastructures and not reinventing the wheel or being inconsistent in between teams (think enterprise). However in my case i'm not sure I do want to build this whole architecture and layers above a KV, my tendency is currently, If I have a KV allow the end-user-developers manage by own the dependencies however this leaves too much room for each to come up with his own solution, which option would you choose?

UPDATE: I think for me the best route to go would be under the assumption of a KV store without relationships:

  1. Do not implement generic relationships layer on top of the KV store.
  2. Light DAO layer.
  3. Each logical/service layer which uses the light DAO layer would need to implement the relationships in it's business logic.

As an appendix to that: if a programmer faces an inconsistency (parent points to a child but child does not exist) then he would need to deal with that with his own custom code + Consider an async process to cleanup these inconsistencies.

Thanks.

  • Do you need to lookup children outside of the parent? Option 3 is appealing if you are always going to load the whole family – Caleth Sep 26 '17 at 8:36
  • @Caleth Almost everything is within the scenario range, in this case, specifically yes children can be looked up separately. – Jas Sep 26 '17 at 9:02
  • 4
    " In elasticsearch you have the notion of parent-child relationships . When we move to a standard key value store we don't have these." if the parent-chiled relationship is an important feature for your application, why don't you choose a database supporting it in the first place? – Timothy Truckle Sep 26 '17 at 9:09
  • @TimothyTruckle awesome comment, although i'm not in control of choosing which db to use, let me take the hat of supporting a plain KV store, the issue is that in that case I prefer to have the speed of a KV store for a set of data which both includes this data with parent-child relationship and also other data which does not need this relationship. (many kinds of data, can't choose different db for each kind of data, too much operations overhead, and speed is the topmost required feature out of this db). – Jas Sep 26 '17 at 9:21
  • 1
    The reasons not to use ElasticSearch as a primary DB likely apply to many NoSQL solutions. Particularly the very loose consistency guarantees mean you have no "source of truth" and even correct functioning of the system can lead to significant, even catastrophic, data loss. From your comments, it sounds you simply have different kinds of data with different needs. In the current data storage landscape, you will typically have systems that consist of multiple technologies. (1/2) – Derek Elkins Sep 26 '17 at 11:37
2

Option 8: maintain the parent child relationship, but mark the child as deleted by setting a deleted property to true inside of its value object.

That one has my preference, because it is a reversible process which happens entirely at the child level (meaning it is atomic).

To cover your bases with a generic KV store you have to take into account that many KV stores are distributed, which means you cannot rely on the order of events being predictable. You may write the child and then update the parent to point to the child, and think you have covered all bases, but a reading process may then retrieve the parent with child pointer and still not be able to find the child yet. In that case the programmer must always handle the "no child exists" scenario. Scenarios where you write to both parent and child often cannot be done atomically and therefore will produce out-of-order behavior.

The programmer that writes the code that interacts with a distributed KV store must be aware of the internal mechanics of that store and deal with the edge cases. An API abstraction that successfully hides these mechanics will not offer a KV interface at all, but instead deal in domain objects.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.