I'm developing a NodeJS API that will be consumed, for now, by an Android app.

I need 2 important things here: a News Feed and a Notification system. And I need it to be scalable.

I'm using MongoDB with Express for this project.

Notification System

About the notification system I'm thinking about a document that would hold:

  • actor: id refering to an User object
  • verb: a String defining the type of action: add friend, comment, etc.
  • object: an id refering to: User, Activity, whatever, depending on the type of verb. I would get the correct object depending on the value of the verb.

My biggest doubt about this structure is: How would I return all this data to my Android app. Is it correct to return a JSON with the notification unpopulated (only ID's), and the Android app would use the ID's and query the API to build the notification. I mean, the client would make a call to: /getUser/:id to get the username, then /getActivity/:id to get the activity details, you get the point... Wouldn't that be to much calls just for a notification?

Because, imagine a user getting 50 notifications, that would be ~100 requests only to get the notification details. If that happens to 1K users, it would be 10K requests to get notification details.

News Feed

What's the best way to my client to get the latest activity? Just query the database for the latest inserted data that matches the criteria?

I shouldn't be dwelling too much on that because the application will start small, but I would like to start with a not so bad design. That's my first application of this type so I don't know what to expect.



Notification Systems

Typically, notification systems use a message queue rather than joins between modelled properties.

In this model:

  • A message queue is just a list you agree to append to.

  • Every user has a unique message queue.

  • Messages for that user are appended to the end of the queue.

  • Calls for new notifications fetch all items in a queue after a specific index, typically in batches of N.

  • Calls for notifications can also include requests between any two indices. This allows you to fetch older notifications.

  • Notification systems typically require persistence. i.e. you don't want older messages to not be retrievable. So events are never removed from the queue.

You're not wrong to model the users of your system with IDs or the type of interactions they can have as strings in MongDB.

The only problem is you're attempting to do a massive outer join to fetch every new event. This, as you've correctly predicted, leads to a large number of requests being made.

There is nothing preventing you from having your backend create new Event objects out of users and interaction types, storing them in a list that's uniquely paired to a user, and consuming them from the queue. This reduces the number of calls you need to make to fetch new notifications significantly.

You might want to look at Apache Kafka, which is an immutable log system that has inbuilt persistence guarantees, rather than a document-oriented database like MongoDB to handle your message queues.

News Feeds

A news feed is just a queue that contains a different event type.

You can reuse message queues here.


You're asking the right questions and have a good starting structure. Have a quick look at W3C Activity Streams 2.0[1], it basically says the same thing: actor, type=verb, object, target (secondary object), and can have other optional data elements. I've often used a 'context' object with a map of relevant properties.

As for the retrieval of notifications, materialize them in separate tables or datastores. I have worked on all scales of systems and this becomes an issue with just hundreds of users. When a user does something the amount of work to write the notification is finite. If you're constantly polling notifications for many users than the load grows with the number of active users whether or not they have new notifications. It also means that you have custom code to synthesize notifications from each type of source.

Now even knowing all that, if you have only a few limited notification types and not many active users, just go ahead and query them if you can develop that faster. Get the features into users' hands. Performance issues are what we like to think about, but until it's a problem, building features and a user base comes first.

[1] https://www.w3.org/TR/activitystreams-core/


Well, in that case :) Check out this repo: https://github.com/GetStream/mongodb-activity-feed It does exactly what you want and even implements aggregation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.