2

I have a news service that has thousands of users.

Each user can subscribe to a set of tags and sites that they individually would like to follow.

A tag or site has articles attached to it.

A user can potentially have 1-500 of these they want to follow.

Each user should have a page where these articles, based on the tags and sites the user follows, are presented.

The update frequency in terms of articles are around 2000 a day.

Every time an article is created/updated/deleted we need to update every user's individual page based on the tags and sites they follow.

I believe we could do a background computation on each users page-data when we update the set of articles. So we populate each user-"bucket" with the data that only that user should be exposed to. But since we do this 2000 times/24h it needs to be performant.

I am uncertain on how this will scale and how it will perform, and I would like some input in regards of how this could be done in a simple yet performant matter.

We use Redis as a data store if that helps you in any way.

6
  • 1
    The "individual page" of each user - is that something you need to create beforehand? Or would it be sufficient to create that "on the fly", after the user gets logged in to the system?
    – Doc Brown
    Nov 22, 2016 at 9:21
  • Sure, if it would perform. The user has a set of id's (tags and sites) and then I need to loop through all articles that are attached to those keys. Not sure it would perform.
    – frostings
    Nov 22, 2016 at 9:22
  • 1
    What did you try? Any measurements so far? I am not sure on Redis, but on most databases, a properly indexed query to find all articles in a specific tag set, with a restriction like "from the last 24 hours" can be very quick. And older articles can be queried "on demand" by the user.
    – Doc Brown
    Nov 22, 2016 at 9:23
  • no, I am only in the "drawing" phase for this.
    – frostings
    Nov 22, 2016 at 9:26
  • 1
    Well, you already decided on Redis, so you are a step further than the "drawing phase". Before you start any premature optimizing, why don't you implement some performance tests first, if you have concerns about it?
    – Doc Brown
    Nov 22, 2016 at 9:51

1 Answer 1

4

My first reaction to these requirements would be to develop a pub/sub system. Every time a change is made to an item, an event would be published to that effect.

Subscribers that care about such events can subscribe to them. Each subscriber can update itself concurrently with all other subscribers, so that sort of architecture ought to scale vertically.

Events should probably go on a queue so that spikes can be smoothed out, and so that subscribers can get updates even if they were online for a while.

1
  • Not sure this fits to the requirements of the OP. A pub/sub system would make sense if the news service shall work as a push service, but from what the OP wrote, it is IMHO not clear if that is what the OP is after.
    – Doc Brown
    Nov 22, 2016 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.