On-the-fly, random partition of a range of numbers [0,N] in groups of max M size

Question

Suppose that you have a range of numbers from 0 to N that you want to randomly separate in groups of max M sequential numbers. This is easily done with a bitmap and a random number generator, but it is not space efficient for very large instances of N.

So, what I want is a generator that, given a number N and M, will start producing non-overlapping groups in a random order and with random size, with less than O(N) space efficiency.

E.g. for N = 7 and M = 3, a valid output is the following:

4
1, 2
3
5, 6, 7
0

Does anyone know an algorithm that can help in producing such an output?

Answer 1

I created a partitioning program with space usage O(1) but running time O(N^2). You can find the source code here. In the comments there is a good explanation of the shuffling algorithm used.

The key part of this program is the shuffling step, which is the step that takes O(N^2) time. Doc Brown asked "how can you shuffle N elements in less than O(N) space"? I extracted the shuffling logic and created a separate program which is listed below.

To get the full explanation, please refer to the source code linked above. The following is a brief explanation:

The shuffling function simulates a Fisher-Yates shuffle, where you swap the array[0] with array[r], where r is a random number in the range [0..N-1]. Then you swap array[1] with array[r], where r is a new random number in the range [1..N-1]. You keep moving down the array, swapping random elements, until you reach the end of the array.

To use O(1) space, there is no array. Instead, for each new random element that we select, we need to replay the previous swaps in backwards order in order to figure out where the array element really came from. In essence, we pick a random element, and then we undo the swaps that came before it to determine where the original position of the element was. We can replay the previous swaps by simply reseeding the random number generator back to a previously saved seed.

Edit: After posting this solution, I found this stackoverflow question which lists some better ways to create a permutation of N numbers in constant space. So if you substitute one of those solutions in for my shuffling function, you can do better than O(N^2) time and still use O(1) space.

/* Given a number N, shuffle the elements from 0..N-1 and print them. */
/* This algorithm uses O(1) space but uses O(N^2) time.               */
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <time.h>

static void shuffle(int n);

int main(int argc, char *argv[])
{
    if (argc < 2) {
        printf("Usage: shuffle N\n");
        exit(0);
    }

    shuffle(atoi(argv[1]));
    return 0;
}

static void shuffle(int n)
{
    uint32_t seedOriginal = time(NULL);
    uint32_t seed         = 0;
    int      i            = 0;
    int      j            = 0;
    int      slot         = 0;

    for (i=0;i<n;i++) {
        seed = seedOriginal;
        srand(seed);

        // Skip n-i-1 random numbers.
        for (j=n-i-1;j>0;j--)
            rand();

        // Select an array slot from [i..n-1].
        slot = i + (rand() % (n - i));

        // Find out what that slot corresponds to in the original order.
        // We do this by backtracking through all the previous steps.
        for (j=i-1;j>=0;j--) {
            int r = j + (rand() % (n - j));

            // Every time we see the slot we are looking for, we switch
            // to looking for slot j instead, because at this previous step
            // we swapped array[j] with array[slot].
            if (r == slot)
                slot = j;
        }

        // Slot is now the correct element we are looking for.
        printf("%d\n", slot);
    }
}

Answer 2

Actually generating random non-overlapping groups in a random order would be a pain in the neck. For example, you'd selected a random "start"; then search existing groups to make sure start is usable and find the "highest possible end" for that start; then select a random "end" that's between "start" and "highest possible end". As long as you're selecting a random "start" you can't avoid some sort of search.

Instead, split it into 2 parts: generate random non-overlapping groups in ascending order; then shuffle them to so they end up in random order.

Generating random non-overlapping groups in ascending order is simple. Each group begins where the previous group ends, and you just need a random length.