0

I have thinking about this idea for over 5 years and i don't have the complete technical knowledge to fully grasp the idea I'm having.

The premise of the idea is to have an extremely high base number system like base 85 / Ascii85.

Then use that high base number to express an extremely verbose seed for a pseudo random data generator.

Then using your data generator, generate out data to a specified length of characters.

The business case for this would hopefully be a way of transferring data in an extremely compressed format that is actually just a function expression of how to calculate out your large set of data.

So for example if you want to move 10 gigabytes of data, you somehow serialize that 10 gigabytes down into a random seed that propagates back out into the same 10 gigabytes again.

The reason for something like Ascii85 is so that you could express a high amount of unique permutations.

This idea is probably completely crazy and or incorrect, but i have wondered about it for so long i really wanted some advice on it.

2

So this is effectively how compression algorithms actually work. More on that in a moment.

The key thing you're missing is the fact that some 85 bit encoding is still just 85 bits. If you have a fixed algorithm, sending an 85 bit seed means you get 85 bits worth of possible permutations. But 10 gigs of data has about 80,000,000,000 bits worth of possible permutations!

But you can think of the compression algorithm as your pseudorandom number generator, and the compressed file itself as the (gigantic) seed. For some input (the compressed file) the algorithm spits out the right sequence of bytes.

"But you just told me it doesn't work! You can't encode 80,000,000,000 bits worth of permutations in less than 80,000,000,000 bits! What's the deal?"

The deal is that compression algorithms guess what your data looks like. They assume it's text (meaning most characters are a-z), or an image (where the background color tends to be uniform) and then encode those patterns with less bits. "Make the next 500 pixels white". They effectively assume that most of those 80,000,000,000 bits worth of permutations won't show up so they don't have to encode them efficiently. Intuitively, that's why re-compressing a compressed file doesn't help - you've already changed it to one of the permutations that the algorithm can't deal with.

0

Yes, this could work. The only problem is this bit

you somehow serialize that 10 gigabytes down into a random seed that propagates back out into the same 10 gigabytes again

is effectively invoking magic. It's also possible that for a given set of data, there is no possible seed to randomly generate it.

  • This answer is a bit misleading, as "possible" here means a non-zero but infinitesimally small possibility. Since you already spent 5 years thinking about this idea, why not spend a few more hours researching the pigeonhole principle and why arbitrary compression of arbitrary data is impossible? – Hans-Martin Mosner Jul 25 '19 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.