1

Lets say there is a real time application where a list is shared among multiple people and updates done to the list must propagate to all the subscribers of the list. Like in trello.

The list is stored in a database on a remote server and list is updated with an API call from the app. The changes are conveyed to the subscribers via a push notification.

The notification may contain the id of the list changed.

Should I send the entire list content to all the subscribers? Or send the details of the changed item? And generally how to ensure that the changes made always reliably reaches all the subscribers, without transferring all the data every time the user logs in?

4

If the list is small and changed frequently, it may be the best to let the subscriber query the full list from time to time. However, if the list is huge and changes occur seldom, you could log any change to the list as a sequence of INSERT/UPDATE/DELETE commands on the server into some kind of commands table. Each command needs a sequence number, in increasing order, and the subscriber should remember which commands were already processed in the local cache. Then, when a subscriber gets a notification that the list "LID" has changed, he sends a query like

SELECT * FROM commands 
  WHERE listid= :LID AND sequenceNumber>:lastSequenceNumber 
  ORDER BY sequenceNumber

to the server and retrieves only the change commands since the former query of that type. Then he processes the commands to the local cache of the list and increases "lastSequenceNumber" accordingly.

Note this will also work when a subscriber misses a notification or with no notifications at all. The subscribers can just do the command querying and processing in regular time intervals, this will still work. The notifications are only for keeping the latency and the required number of roundtrips low.

If you go this route, you will also need a strategy for starting new subscribers to an existing list, with an empty local cache. For this case, the whole list should be transferred to the subscriber once, together with the last sequence number of the final command which was already applied to the list.

If you google for "Command pattern" or "CQRS" or "event sourcing", you will find more information about this strategies.

  • This is what I was looking for.... Thanks a lot... I will try to implement my own version of this and will update but this looks golden as it is. – GriffinTaimer Sep 7 '17 at 19:44
2

The sole purpose of the notification is to notify.

Your notification should enough information to the client so that the client can decide what it must do to update itself, including retrieving any necessary data from the database. It's entirely possible that the ID of the list that has changed is all that the client needs.

Generally speaking, when a subscriber logs in, their client will always retrieve a fresh copy of the data from the database. So unless you maintain some sort of local cache in the client, you shouldn't have to reason about notifications that occur when a subscriber is offline.

  • I will have a DB locally in the app, this was to avoid transferring huge amounts of data when the user logs in. But what if by chance the user misses a notification? Then the data locally will be out of sync with the remote one... – GriffinTaimer Sep 7 '17 at 19:13
  • 1
    Assuming you have logic to bring the local database back in sync, you should always perform such a sync when the user logs in. – Robert Harvey Sep 7 '17 at 19:14
  • He'll be signed in until he logs out manually or deletes the app or clears data. So if he misses the notification when he has signed in, there will be no way for me to obtain the modified data. – GriffinTaimer Sep 7 '17 at 19:17
  • You don't need the notifications to perform a database sync (if you have designed it properly). – Robert Harvey Sep 7 '17 at 19:17
  • 1
    Unless its causing problems like application delays or excessive memory usage (or your clients are on bandwidth-constrained devices like mobile phones), it doesn't matter. – Robert Harvey Sep 7 '17 at 19:53
0

You could store the data in an immutable, persistent datastructure. Now, if you expose that structure to the clients, a sync consists of downloading the new nodes (i.e., nodes that are not shared with the old version). This will involve a number of nodes proportional to the "diff" between the two versions.

This means your notification can be just that — a notification. You could (but don't have to) include a reference to the root of the new structure with the notification. That is sufficient to reference that entire version. The client can then choose precisely what to download.

In addition, a client can initiate a sync without a notification by simply asking the server for the current root. This allows for the case where a client misses a notification for some reason.

Since you describe your data as a "list", you might find something like Relaxed Radix Balanced (RRB) Trees appropriate. You might also draw inspiration from Hinze and Patterson's "Finger trees: a simple general-purpose data structure".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.