0

I'm trying to create a website as a forum similar to Reddit, but on a smaller scale (no comments also). The website will allow users to post a link to a forum. Then they can view a list of New or Trending links based on upvotes (gathered from an external API).

So basically all I need is to have a REST API to post links to one of many long lists, each list then users can view from with GET requests. There also needs to be a quick way to get the last 7 days of links so that the most popular ones can be computed. I'm accommodating for a maximum of about 100,000 links posted daily (about 1 per second) or and maybe 10 requests to Trending and New posts.

My current architecture would be:

  • Stores past n (likely n=7) days of links in memory, then periodically empties that list into a long-term storage file.
  • Then to get New links, generally just queries memory but if a subforum is not very active gets older links from file.
  • Periodically updates the list of Trending links (short list in memory) for each subforum, then to get these Trending links queries that list in memory.

This will be mostly built for learning and it will not reach the 100,000 links daily, but I want to build something that can scale to that level so that I will be prepared if I have to in the future.

Would my above architecture be scalable, is there a better way of doing this, any other suggestions? Thanks for your help ahead of time.

1

Just about any database will be able to handle 1 insert per second especially if its just a link and title. The problems you will face with scaling will largely not be known until you build it. Largely they will be badly written queries or pages that require a lot of requests to extract the needed data. This example site is so simple that you don't even need to put much consideration in to how you structure it. As long as you keep things fairly normal and write half decent queries you would have a hard time not meeting your requirements.

Because there are no complex calculations that need to be cached or any room to reduce the work being done you will probably find that really the only way to be able to scale the speed of your inserts is to just put the database on better hardware or across multiple servers.

If you are receiving requests to view data far more often than it is being submitted you will want to simply cache the current results page and show a cached version to visitors but at 1 submit per second the cache may be being invalidated before its even useful.

  • Ok, I was just making sure I was on the right track. Thanks for the help! – Nicholas Jan 17 at 3:06
  • 1
    @Nicholas I have just one more point. From your post you did not mention a database. Do not attempt to manage all this data in some data structure in memory and dump it to a text file. Every single tool and framework supports this easily so read up on how its done but you will save yourself so much trouble this way. – Qwertie Jan 17 at 3:15
  • So actually I simplified the question a little bit; what I'm really trying to do is create a p2p node which keeps track of a large number of requests. I want it to be fairly to run on developers wanting to build using its' computer and don't know of any database that can be easy to set up as quickly as downloading a release, configuring a few environment variables, and running. – Nicholas Jan 18 at 2:55
  • Thanks for the help, I've decided to use sqlite to handle this, as something like mongo seems like overkill – Nicholas Jan 19 at 23:05

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.