It feels like your question is very related to your previous stackoverflow question.

Generally speaking, Azure Table IO performance improves as more partitions are used (with some tradeoffs in continuation tokens and batch updates I won't go into).

 

Since the partition key is always a string I am considering using a "natural" load balancing technique based on a subset of the GetHashCode() of the partition key, and appending this subset to the partition key itself. This will allow all direct PK/RK queries to be computed with little overhead and with ease. Batch updates may just need an intermediate to group similar PKs together prior to submission

With that background information the problem is not simply taking a database of words and splitting into multiple smaller but equal sized databases of the same words. Instead it looks like you are trying to determine how to automatically partition your data based on a key so that it is separated into equal parts.

If our end goal is to partition the words evenly across the shards I would not do so based on the data itself. That is to say if my method for determining which partition a bit of data goes into is based on what is included in that data I would not expect my end distribution to be even.

Given the follow list of words as an example

Programmer,Programmer,StackExchange

The value Programmer is repeated twice. If that value is what I am using to determine which partition the data will end up in I would expect that both Programmers would end up in the same partition. If the end goal is to keep all closely related words on the same partition this may be the way you want to go, but I don't think it is.

If instead we do not care which words are where and want an even distribution I would first determine the number of shards that I want to run and then simply loop through my database of words assigning the partition key 1-N where N is the number of partitions I have.

i.e.

int totalPartitions = 5;
int currentPartition = 1;
Foreach(var item in MyData) {
  MyData.PartitionKey = currentPartition;
  if(currentPartition < totalPartitions)
    currentPartition++
  else
    currentPartition = 1;
}

It feels like your question is very related to your previous stackoverflow question.

Generally speaking, Azure Table IO performance improves as more partitions are used (with some tradeoffs in continuation tokens and batch updates I won't go into).

Since the partition key is always a string I am considering using a "natural" load balancing technique based on a subset of the GetHashCode() of the partition key, and appending this subset to the partition key itself. This will allow all direct PK/RK queries to be computed with little overhead and with ease. Batch updates may just need an intermediate to group similar PKs together prior to submission

With that background information the problem is not simply taking a database of words and splitting into multiple smaller but equal sized databases of the same words. Instead it looks like you are trying to determine how to automatically partition your data based on a key so that it is separated into equal parts.

If our end goal is to partition the words evenly across the shards I would not do so based on the data itself. That is to say if my method for determining which partition a bit of data goes into is based on what is included in that data I would not expect my end distribution to be even.

Given the follow list of words as an example

Programmer,Programmer,StackExchange

The value Programmer is repeated twice. If that value is what I am using to determine which partition the data will end up in I would expect that both Programmers would end up in the same partition. If the end goal is to keep all closely related words on the same partition this may be the way you want to go, but I don't think it is.

If instead we do not care which words are where and want an even distribution I would first determine the number of shards that I want to run and then simply loop through my database of words assigning the partition key 1-N where N is the number of partitions I have.

i.e.

int totalPartitions = 5;
int currentPartition = 1;
Foreach(var item in MyData) {
  MyData.PartitionKey = currentPartition;
  if(currentPartition < totalPartitions)
    currentPartition++
  else
    currentPartition = 1;
}

It feels like your question is very related to your previous stackoverflow question.

Generally speaking, Azure Table IO performance improves as more partitions are used (with some tradeoffs in continuation tokens and batch updates I won't go into).

Since the partition key is always a string I am considering using a "natural" load balancing technique based on a subset of the GetHashCode() of the partition key, and appending this subset to the partition key itself. This will allow all direct PK/RK queries to be computed with little overhead and with ease. Batch updates may just need an intermediate to group similar PKs together prior to submission

With that background information the problem is not simply taking a database of words and splitting into multiple smaller but equal sized databases of the same words. Instead it looks like you are trying to determine how to automatically partition your data based on a key so that it is separated into equal parts.

If our end goal is to partition the words evenly across the shards I would not do so based on the data itself. That is to say if my method for determining which partition a bit of data goes into is based on what is included in that data I would not expect my end distribution to be even.

Given the follow list of words as an example

Programmer,Programmer,StackExchange

The value Programmer is repeated twice. If that value is what I am using to determine which partition the data will end up in I would expect that both Programmers would end up in the same partition. If the end goal is to keep all closely related words on the same partition this may be the way you want to go, but I don't think it is.

If instead we do not care which words are where and want an even distribution I would first determine the number of shards that I want to run and then simply loop through my database of words assigning the partition key 1-N where N is the number of partitions I have.

i.e.

int totalPartitions = 5;
int currentPartition = 1;
Foreach(var item in MyData) {
  MyData.PartitionKey = currentPartition;
  if(currentPartition < totalPartitions)
    currentPartition++
  else
    currentPartition = 1;
}

It feels like your question is very related to your previous stackoverflow question.

Generally speaking, Azure Table IO performance improves as more partitions are used (with some tradeoffs in continuation tokens and batch updates I won't go into).

Since the partition key is always a string I am considering using a "natural" load balancing technique based on a subset of the GetHashCode() of the partition key, and appending this subset to the partition key itself. This will allow all direct PK/RK queries to be computed with little overhead and with ease. Batch updates may just need an intermediate to group similar PKs together prior to submission

With that background information the problem is not simply taking a database of words and splitting into multiple smaller but equal sized databases of the same words. Instead it looks like you are trying to determine how to automatically partition your data based on a key so that it is separated into equal parts.

If our end goal is to partition the words evenly across the shards I would not do so based on the data itself. That is to say if my method for determining which partition a bit of data goes into is based on what is included in that data I would not expect my end distribution to be even.

Given the follow list of words as an example

Programmer,Programmer,StackExchange

The value Programmer is repeated twice. If that value is what I am using to determine which partition the data will end up in I would expect that both Programmers would end up in the same partition. If the end goal is to keep all closely related words on the same partition this may be the way you want to go, but I don't think it is.

If instead we do not care which words are where and want an even distribution I would first determine the number of shards that I want to run and then simply loop through my database of words assigning the partition key 1-N where N is the number of partitions I have.

i.e.

int totalPartitions = 5;
int currentPartition = 1;
Foreach(var item in MyData) {
  MyData.PartitionKey = currentPartition;
  if(currentPartition < totalPartitions)
    currentPartition++
  else
    currentPartition = 1;
}