0

When does it make sense to put data in elastic search vs creating secondary indexing on Primary datastore? Elastic search with another primary store

Pros:

  1. Primary datastore can be optimised for read write usecases.
  2. Elastic search suports more than just key value matching like fuzzy match, etc.

Cons:

  1. Out of sync with primary datastore
  2. two more component to manage (ES as well a pipeline to insert in ES)
  3. Would need some sort of Change data capture capability from Primary datastore.

Secondary indexes on Primary datastore

Pros:

  1. Less moving parts.
  2. Less consistency issues ( because secondary indexes can be eventually consistant)

Cons

  1. Not all datastore support secondary indexing
  2. Secondary index queries are more oftan scatter gather, doing it on higher QPS will limit read write qps on primary access patterns like read, write by PK

Are there other considerations while deciding this?

1 Answer 1

1

I was given good advice by an architect, early in my career: design and extra complexity, to improve performance, before you’ve discovered where the actual bottle decks in the system are. Otherwise, you will likely end up optimizing the wrong area.

I’d recommend trying this out with full (fake?) volume on your primary data store with a secondary index to determine if that beat your performance requirements and is the actual bottleneck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.