How do you design a website that allows users to query a large amount of user data, more specifically:
- there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database)
- number of (concurrent) queries is not important, but each query should be as fast as possible
- support some simple queries such as: get user info by id, get accumulated data like monthly logins and monthly online time
- query result is little (1 number, or a few hundred rows) so frontend performance doesn't matter
I'm more interested in the thought process on how to approach this requirement. For example:
- at 100 users, what is the design?
- at 1,000,000 users, what needs to be changed?
- at 100,000,000 users, what is the design now?
I've searched around and see a lot of people talking about caching, load balancing,... Of course, those techniques are useful and can be used but how do you know it can help handling N users? Nobody seems to explain this point.