0

I am looking for a way of dealing with the following situation:

1- Have an items collection in MongoDB

2- Have a users collection in DynamoDB

3- Each document in the items collection has a "reference" to a user or multiple users via an id attribute of the DynamoDB.

4- I need to query the items collection and populate each item with metadata from the user table (In each item, I have to replace the id reference/s with user metadata)

5- The DynamoDB has to remain as the main source of truth for the users, and can't move the MongoDB data over.

I'm wondering what would be the best solution for this.

At the moment I'm thinking of two options:

  • One is getting the items from MongoDB, getting the users ids, query all the users from DynamoDB and inserting the user's data in the items. As the items are somewhat a complex collection, there is a lot of potential to introduce new bugs and making it hard to maintain in the future.

  • Second one is copying the DynamoDB users to MongoDB and querying directly in MongoDB. The issue here is that the collection has to be kept in sync all the time and I'm not convinced about having the same data in different places due to the increased complexity.

As no approach is 100% perfect, just wondering if there could be a better way to do this.

Which option (or other) would be the way to go?

Improve this question
5
  • thanks, @DocBrown, I just edited it to make it more clear to I need to query the items collection and populate each item with metadata from the user table. In each item, I have to replace the id reference with user metadata.
    – alvm
    Commented Sep 24, 2021 at 13:27
  • Thanks. Another thing I don't understand is why, in your first approach, the complexity of items has more potential to introduce bugs than your second. I mean, you wrote, regardless of the approach you choose, you have to "populate each item with metadata from the user table" - does "populate each item" mean something different to you than "insert the user's data in the items"?
    – Doc Brown
    Commented Sep 24, 2021 at 13:53
  • @DocBrown No, it means basically that. But is not as straightforward, in any given document of an item you could have several references to a user(s), nested within objects or an array of objects. There is a reasonable risk of missing edge cases that could potentially introduce issues. Also, you could get one item or N items, were getting N items if N is large enough, the performance could take a hit as you have to iterate every item/property and insert the user(s) whenever is needed.
    – alvm
    Commented Sep 24, 2021 at 14:11
  • Ok, but it seems in both approaches, you get user data from the DynamoDB and store it inside the MongoDB, right?`In the first approach, you store it inside the item documents directly, in the second, you store it in a separate user collection. In both cases, you have to make sure the user data is up-to-date in the MongoDB, so where is the difference here?
    – Doc Brown
    Commented Sep 24, 2021 at 14:37
  • @DocBrown, no, the items collection only have a reference with an Id of the user. The population of the users, in this case, is done when a read request is made. The flow when a get request arrives: get items with users id from MongoDB -> query dynamo DB for users metadata -> iterate and populate items with this user metadata data -> return response items with users metadata.
    – alvm
    Commented Sep 24, 2021 at 22:31

1 Answer 1

Reset to default
0

About the first approach you suggested, the developers are responsible for the document structure, and if a new key is added to the structure with user id, they should query it too, and if not they are responisble for this bug. (Also in the second approach it will happen)

Pros:

  1. you dont need to sync data between databases
  2. the users info will be updated (in the second approach it will be eventually consistent with some delay)

Cons:

  1. each query is depending on two databases, but its not really issue and it can be solved by cloning the dbs

About the second, you should use a queue like kafka/rabbitmq and transfer updates on the dynamodb to mongodb.

Pros:

  1. If dynamodb has a failover the data is still accessible

Cons:

  1. there can be a delay of the current users data
  2. Adding complexity to the architecture
  3. The sync mechanism adds a lot of calls for both dbs

I think the first approach is better.

But i dont think the architecture is good enough.. seems like one relational db is classic for here (but i dont know the full details and constraints)

Improve this answer
1
  • Yes, thanks for your time. I am a bit constrained, but this gives me ideas of how to approach it!
    – alvm
    Commented Nov 1, 2021 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.