For access to my API user should send login and password and get generated token for access.


Suppose the size of account's table is very large. So large - so needs sharding. Usually I prefer sharding by ranges (which excludes re-sharding) when I do it for tables with primary key ID (and the same shard key). But in this case sharding key should be a string (login).

What is the last best practices you use for this task? Consistent hashing (from Guava, for example) and resharding only (K / n) keys in the case of changing structure of nodes? May be there are some practices without re-sharding? Something else?

  • 1
    Are you creating a social network for insects, or why do you have so many users that you need to shard this? – CodesInChaos Aug 10 '15 at 16:12

If you do have to go with re-sharding, one way to do this is to use another database table to store the actual account and host/table mapping. Mapping table contains only key fields to lookup on which are optimized for performance (id, name, email) and rest of the account data is in the table that the mapping table points to. This gives the most flexible re-sharding technique.


Just how many users are you expecting to have?

I've worked with systems that have hundreds of thousands of users and have never felt the need for sharding (just proper indexing).

And, of course, you're not actually storing the password, are you ...

  • 10^9. I have added 10^9 users to my developer's postgres (only 8 GB RAM, but after pg_tune) and it is quick. But the question is how many requests should serve this single database in a second? I don't know. What is the peak? When I try to request authentication (select by login with unique index for 10^9 records) in parallel with 40 threads then it obviously slow down. In the case of sharding it will be quick. I will split it on virtual buckets (tables or schemas) and only after that I will move some of them to separate servers (if necessary). – Ivan Jun 15 '15 at 8:00
  • Of course, I don't store password in a plain text. You can read name of this field as encrypted_password. It contains SHA-256(password + salt). – Ivan Jun 15 '15 at 8:00
  • 1
    @Ivan 10^9 users. That is like 1/7 the world population. How many authentications do you need to support a second? – paparazzo Aug 10 '15 at 13:08
  • @Ivan 1) With logins the computation of the password hash if by far the most expensive step, and that step happens on the application side and is thus can be distributed trivially among several servers. 2) SHA-256(password + salt) an insecure hash, since it's too cheap to compute. See How to securely hash passwords? for details. – CodesInChaos Aug 10 '15 at 16:13
  • @CodesInChaos, I didn't know about SHA-256, thank you. I'll migrate on BCrypt. – Ivan Aug 12 '15 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.