2

I have written a basic web app in PHP, using MongoDB as database engine. The app basically inserts records into database as they become available, and queries the database and displays the data as on a web page when the user visits the page.

When the database is queried and cursor is returned, some processing is done on each record, like checking the age of an individual and deciding which age-category (like under-age, over-aged, just-the-right-age ;) ) he falls into, and then the data is displayed on the web page.

My Question: Previously I developed this system for 500 database records. It worked fine. BUT now I have to develop this system for 1 million or more database records.

So what changes should I make to the existing system, to make it ready for that big data?

  • 5
    That's a pretty vague problem statement. And, "1 million records" isn't inherently problematic. What indication do you have that there's a potential problem? Do you have benchmarks? ... And where are a million records going to suddenly come from? – svidgen Jan 14 '18 at 5:41
  • 1
    Have you tried dumping 1million documents into mongoDB and see how the system behave? – Laiv Jan 14 '18 at 10:53
  • The critical thing in dealing with large databases is making sure that every query you use has adequate indexing in the tables so that it doesn't need to perform a scan over all the data. Any query that has to scan everything will kill your performance at scale. I'm not familiar with how mongodb manages such things, but there should be a way of asking it what operations it performs to respond to a particular query, that should show whether it touches the whole dataset or not (in mysql, for example, you'd use an EXPLAIN SELECT query to check this). – Jules Jan 15 '18 at 0:26
  • @Jules Of course, "less than a thousand" users and "1 million records" do not describe a "large database" or working "at scale". Any database technology will perform adequately at this level with little to no special effort taken. – Derek Elkins Jan 15 '18 at 1:28
  • @DerekElkins 1 million records is more than enough to cause significant delays if inadequate indexing is used. – Jules Jan 15 '18 at 1:32
5

OK, so answering your question literally, You don't need to make any changes.

The volume of data on the database isn't usually a scaling issue for webpages. Any given webpage will only be looking at a small part of the data and databases are designed to retrieve subsets of data from large sets very quickly.

What you need to take account of is how quickly you can generate a page. This is usually limited the number of requests per second for webpages that your server is receiving. Each one takes up a chunk of CPU time and your server has a limited amount. Once you hit 100% every page request will get slower and slower.

So its not the million rows you have to worry about, its the million users.

Website servers are cheap to scale, you and fire up webserver2, copy your website to it and double your capacity with no major technical hurdles.

However, both websites will now be using the same database server. and once that hits capacity you do have some significant techincal problems to solve.

The underlying problem is that you want all the database requests to be looking at the same data. If you simply copy it across to a second db the databases will quickly go out of sync.

If you continually update each with data from the other, the each server is doing twice as much work and you haven't solved your scalablity problem.

MongoDB and other nosql databases are designed to combat this by ignoring it. instead of having a number of tables which must all be consistent, you put all your data into one blob. so its always consistent with itself.

This allows you to spin up multiple instances and just copy the data without too much of an issue. It then solves the problem of the blobs being updated with some clever tricks you don't have to worry about too much.

So in summary: You current php + mongodb website, assuming you haven't made any 'gothcha' errors should scale up to any number of users by just throwing more servers at the problem. If you are cloud hosting that just means clicking a button or two and putting your credit card in.

  • The number of users will be less than a thousand. SO that is not a problem. But DATA, on the other hand, will keep growing. New data will keep becoming available forever, and it will keep being dumped into the database. So given this info, an advice for me? – JBel Jan 14 '18 at 17:02
  • @JBel: Databases are designed to store large (really large) amounts of data. As long as you don't try to retrieve it all at the same time, you are fine. – Bart van Ingen Schenau Jan 14 '18 at 17:56
  • @jbel you really need to give more info. obviously there are various problems to be dealt with on the road, but this is what the products are designed to do. Untill you hit terrabytes there shouldnt be anything the standard processes cant deal with – Ewan Jan 14 '18 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.