Designing a Feed and Notification system in MongoDB

Question

I'm developing a NodeJS API that will be consumed, for now, by an Android app.

I need 2 important things here: a News Feed and a Notification system. And I need it to be scalable.

I'm using MongoDB with Express for this project.

Notification System

About the notification system I'm thinking about a document that would hold:

• actor: id refering to an User object
• verb: a String defining the type of action: add friend, comment, etc.
• object: an id refering to: User, Activity, whatever, depending on the type of verb. I would get the correct object depending on the value of the verb.

My biggest doubt about this structure is: How would I return all this data to my Android app. Is it correct to return a JSON with the notification unpopulated (only ID's), and the Android app would use the ID's and query the API to build the notification. I mean, the client would make a call to: /getUser/:id to get the username, then /getActivity/:id to get the activity details, you get the point... Wouldn't that be to much calls just for a notification?

Because, imagine a user getting 50 notifications, that would be ~100 requests only to get the notification details. If that happens to 1K users, it would be 10K requests to get notification details.

News Feed

What's the best way to my client to get the latest activity? Just query the database for the latest inserted data that matches the criteria?

I shouldn't be dwelling too much on that because the application will start small, but I would like to start with a not so bad design. That's my first application of this type so I don't know what to expect.

Thanks!

Answer 1

Notification Systems

Typically, notification systems use a message queue rather than joins between modelled properties.

In this model:

• A message queue is just a list you agree to append to.

• Every user has a unique message queue.

• Messages for that user are appended to the end of the queue.

• Calls for new notifications fetch all items in a queue after a specific index, typically in batches of N.

• Calls for notifications can also include requests between any two indices. This allows you to fetch older notifications.

• Notification systems typically require persistence. i.e. you don't want older messages to not be retrievable. So events are never removed from the queue.

You're not wrong to model the users of your system with IDs or the type of interactions they can have as strings in MongDB.

The only problem is you're attempting to do a massive outer join to fetch every new event. This, as you've correctly predicted, leads to a large number of requests being made.

There is nothing preventing you from having your backend create new Event objects out of users and interaction types, storing them in a list that's uniquely paired to a user, and consuming them from the queue. This reduces the number of calls you need to make to fetch new notifications significantly.

You might want to look at Apache Kafka, which is an immutable log system that has inbuilt persistence guarantees, rather than a document-oriented database like MongoDB to handle your message queues.

News Feeds

A news feed is just a queue that contains a different event type.

You can reuse message queues here.

Answer 2

You're asking the right questions and have a good starting structure. Have a quick look at W3C Activity Streams 2.0[1], it basically says the same thing: actor, type=verb, object, target (secondary object), and can have other optional data elements. I've often used a 'context' object with a map of relevant properties.

As for the retrieval of notifications, materialize them in separate tables or datastores. I have worked on all scales of systems and this becomes an issue with just hundreds of users. When a user does something the amount of work to write the notification is finite. If you're constantly polling notifications for many users than the load grows with the number of active users whether or not they have new notifications. It also means that you have custom code to synthesize notifications from each type of source.

Now even knowing all that, if you have only a few limited notification types and not many active users, just go ahead and query them if you can develop that faster. Get the features into users' hands. Performance issues are what we like to think about, but until it's a problem, building features and a user base comes first.

[1] https://www.w3.org/TR/activitystreams-core/

Answer 3

My biggest doubt about this structure is: How would I return all this data to my Android app. Is it correct to return a JSON with the notification unpopulated (only ID's), and the Android app would use the ID's and query the API to build the notification. I mean, the client would make a call to: /getUser/:id to get the username, then /getActivity/:id to get the activity details, you get the point... Wouldn't that be to much calls just for a notification?

That's why your documents should have all the information instead of references to different collections. Even if it means duplicating information. It doesn't matter. If the original "feed" is edited, then a new "Message" is created (or the older one is updated) and "published".

MongoDB "supports" joins through lookups and newer versions of MongoDB might support SQL but whether you can take advantage of these features will depend on the specific programming language drivers and the specific MongoDB version! Something I have learnt the hard way. Upgrading the DB version was no longer as easy as promoting to the next version. It takes a lot of tests to ensure that the new version doesn't cause breaking changes and the new drivers either. The features you are coupled with must behave exactly the same way and the query operators have the very same contract and format. That's something you don't have to care about with most of the RDBMS. But with MongoDB, you must.

For example, MongoDB supports different Aggregation operations, but it takes having the latest java driver to be able to use'em all programmatically. If you design your documents in such a way your queries depend purely on core features it's unlikely changing the MongoDB version will have any impact on your source code or dependencies.

I can't tell you what specific model you should return to the Android client, but in any case, regardless of how data is persisted, you can return a projection with the bare minimum information to make the message readable. So, use lookup and projections to compose specific MongoDB response models and map them to DTOs.

If you have to fetch more information from other sources, do it on the server-side and complete the DTO you obtained in the first place. That's the whole point of DTOs.

If you return a snapshot of the message, the user has something to read and enough information to decide whether it is worth reading the whole message (which will trigger a second request) or just ignore it.

What's the best way for my client to get the latest activity? Just query the database for the latest inserted data that matches the criteria?

No silver bullets or shortcuts here. It's an architectural decision you should make based on your needs. You can re-invent the wheel with MongoDB and your programming language or you can use existing solutions. Essentially you will look for solutions supporting pub-sub patterns. Note that, you don't have to code this in the same backend. You can promote this feature and make it a standalone service, written (or not) in a different language. Whatever takes less effort to maintain.

Answer 4

Well, in that case :) Check out this repo: https://github.com/GetStream/mongodb-activity-feed It does exactly what you want and even implements aggregation.