I have following entities:
- Categories: Interests, Disinterests, Categories A, B C...
- Each categories have sub categories: Interests=Gaming, Physics, Programming etc..
- Now each user could belong to multiple sub categories.
- So there is M:N relation ship between categories and users.
- 1 Billion Users
- 100 Categories each could have subcategories ranging form 100 - 10,000
Batch Read and Write: Selection and Projection given user_id. E.g. Get all the Interests of user A.
Real Time Read and Write: I need to get all the users for a given field like Interest:Games.
I used separate files for each sub categories containing list of users. Redis server has keys-value pairs like
However this design has many limitations, like slow access times due to disk operations to get all users for a given category. Huge number of keys in Redis i.e. number of (users * number of sub_categories).
I need a change of design
Current plan is to use MongoDb to maintain hierarchical data for user <-> categories mapping.
<User_id, Interests, A, B, C>.
Each categories will have children fields. Since MongoDB is in-memory DB access using user_id should be faster right? But how about reverse query where I specify Interest::Programming as key? Is there any better way I could design it?