0

Looking at sharding techniques, you basically have hash-based or range-based versions. Hash based is more random, range-based is more heuristic based. Say you initially have 2 shards (separate database instances), then you outgrow it and would require 3 shards. What do you do, now your hash stuff won't work anymore, you basically need to recompute the position of every record (say there are 10 billion records). And range sharding won't transfer either, you'd have to give more weight potentially to the new shard, but that wouldn't really be ideal either so in this case you would also have to recompute record locations.

Am I thinking about this correctly? Then say at your current growth rate, you should ideally add 2 more shards for a total of 5, then 3 more for a total of 8. Each time you would have to re-scatter all the records, correct?

If so, what happens in practice when you need to do this readjustment? I have never dealt with databases needing this level of complexity. Is the database usable during this time? How do you both transfer the records to the new sharding scheme, and at the same time allow your database to be read from and written to?

Improve this question
3
  • I've never done this before, but my instinct is telling me the application will be down while data is reorganized, and a node is added.
    – Greg Burghardt
    Jan 23, 2022 at 13:15
  • You're looking for the concept of consistent hashing which reduces the number of buckets that must be moved between shards when adding or removing shards. It is in principle possible to add a shard without downtime because the old shards can still serve clients while the new shard is being initialized.
    – amon
    Jan 23, 2022 at 15:07
  • @GregBurghardt: It may be possible to temporarily redirect the incoming writes to a queue, leaving the application functionally online, except that the reads won't contain the updates (or still do, but then you have to run the old system in its entirety, reshard on amother system, and queue the commands as well). It's resource-intensive but possible if warranted.
    – Flater
    Jan 23, 2022 at 23:09

1 Answer 1

Reset to default
4

One approach that works independent on what sharding rules you use can be following: Use double checks during resharding.

After you added new nodes, start resharding, one node at a time. The node should iterate over all its elements and apply new sharding rules. Some elements will be moved to other shards, some will remain. Until all nodes resharded, when a request received, the system should do a double check: First determine the node based on new sharding rules and send request to it. If not successful, determine node based on the old sharding rules and send request to it.

Advantages:

  • No downtime. This can be especially important when nodes contain a lot of data and resharding takes a lot of time.
  • It works for any sharding rules.
  • It is very robust. In case the process was interrupted because of some reason, you can start it from the very begin, not making any assumptions about how far the process was before interruption.

Disadvantages:

  • This requires processing of every element and thus may be not the fastest one.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.