Some Guidance on Parent-Child Relationships in Elasticsearch

Question

I would like some guidance in setting up my document structure Elasticsearch. The company I work for has an app that stores around 20,000 new phone records each day in a SQL database. And we feel we could benefit from the features that Elasticsearch provides (fast searching). So we are going to start storing these records in an Elasticsearch database, and provide some search tools that will leverage this as well.

I'm new to NoSQL and Elasticsearch (my background is SQL). So I've read some tutorials on Elasticsearch (and NoSQL in general) to get me started on this project. And I think that I understand (from a general perspective) how to go about this. The basic structure I want to set up is this:

--Owner (ownerId)
  --Call (callId, to, from, location)
  --Call (callId, to, from, location)
  --Call (callId, to, from, location)
--Owner (ownerId)
  --Call (callId, to, from, location)
  --Call (callId, to, from, location)
  --Call (callId, to, from, location)

It's really straightforward. Each business (business=owner) has many calls that we store. And each call also has some additional info (that will be search-able), such as caller names, phone numbers, locations. Also, each owner has a unique identifier, and each call has a unique identifier. Also, calls from different owners never cross-pollinate. We never have a reason to join them.

Upon reading some docs and watching some tutorials, I see 3 possible ways I could implement this structure. I would like some guidance (from seasoned Elasticsearch developers) on these 3 methods. I feel confident that each method would work. But I'm not sure what the long-term implications might be. I need to make sure the method I use supports a high volume of adds and updates (20,000 each day), and still provides fast searching. It also needs to support the initial import of our existing data (we have around 4 million call records already in the database).

Here are the 3 methods:

1. Parent-Child Relationship (some info found here, among others).
2. Utilize one call store for all owners, and store the ownerId in each Call document. When creating the search interface, I would add the ownerId as a hidden search param to make sure the user only retrieves calls for his/her business. (Note: this resembles a one-to-many relationship created by a FK in traditional SQL)
3. Create a separate document store for each owner. For this one, I would create a separate call store for each owner. I would append the ownerId to the name of the document store. So I would have a store called calls_8j934jok83 and another one called call_98ged34h2, and so on (8j934jok83 and 98ged34h2 are separate ownerIds.

Having no experience in NoSQL or Elasticsearch, I feel like I am playing with fire here. Like I said, I feel that any of these three would "word". But I want to make sure I use best practices, and that the structure I set up will be scale-able (we have around 20,000 calls coming in each day now, but we are growing).

Answer 1

Method 2 Sounds Best

It's simplest and seems most suited to the situation you describe. It has been over a year since I worked on Elasticsearch. This is what I recall.

Why? Method 1 vs. Method 2

If I had to pick between these with no further knowledge I would pick Method 2. It's hard to see why a parent-child relationship is beneficial if you are going to restrict searches to one Owner at a time.

If you are drawn to the parent-child relationship, you should try it before committing. It makes every query a little more complicated, and the results may not always be what you expect.

Also consider the parent-child relationship's effect of restricting an owner's data (and searches) to a single shard. This could be harmless for your scale and data. Or it could be a disadvantage, defeating Elasticsearch's ability to execute a single search request on >1 node in parallel for your largest Owners. Consider whether your largest Owner will become a hotspot that suffers from being restricted to a single node, or creates a busy shard that will affect some other Owners with data on that same shard and its replicas.

The parent-child relationship does not sound right for the situation you described.

Why? Method 3

This sounds potentially quite bad, depending what you mean by 'document store'. If you mean a separate Index for each Owner, consider how many Owners this is likely to be and how that will affect your infrastructure requirements. Elasticsearch can handle a lot of Indexes but there is a cost. Each index has Shards and Replicas. Each Shard or Replica takes some resources on a Node.

If you are expecting a few owners the cost may be insignificant and you can configure each Owner's index appropriately. If you are expecting thousands of owners the cost may be prohibitive.

If you mean a separate Cluster for each owner, this would be the most expensive way to go, and would probably be justified only by demanding data segregation requirements (e.g. HIPAA/HITECH).

Try Sample Searches Before You Commit

Before you pick a style, make sure you understand how you will have to write queries and whether that works for your purposes. Experience with databases does not necessarily give you good intuition for the results of Elasticsearch queries.

I used to load some sample data and do searches through Kibana console as a quick way to get a feel for it.

For the simpler style (Method 2), make sure you index the ownerId field in a way that allows you to specify an exact match. You don't want two Owners coming back in results where you thought you specified just one.