1

I'm working on a project with huge files that contain only the set {[0-9],.}. Encoding in UTF-8 or ASCII make huge files.

I wonder if I could find a way to encode in only 4 bits (make those file 16 times smaller). I would have enough of 16 characters!

How could I proceed? I'm working on python and I have tried the following code. Seems like the computer doesn't want it, because it only erases what was in the text file...

def encodeData(self, filePath):
    print("File size before 4bits Encoding = {} Bits" .format(os.path.getsize(filePath)))
    print("Proceed with '{}' 4 bit Encoding... ".format(filePath))
    with open(filePath, "w+") as file2encode:
        for line in file2encode.readlines():
            for i in len(line):
                line[i] = 0x4
    print("File size After 4bits Encoding = {} Bits".format(os.path.getsize(filePath)))

Is there a way I can play with 4 bit encoding?

  • You need to start thinking in terms of a binary format, and using bit manipulation operators. – whatsisname Nov 14 '18 at 3:32
  • Alright! could you elaborate? how do I storage binary data ? – PyThagoras Nov 14 '18 at 3:38
  • you should use with open(filePath, "r") as f: first to read from the the file, then use w+ or wb(write as bytes) to write back, so it won't "only erases what was in the text file" – Aries_is_there Nov 14 '18 at 5:33
  • 1
  • Im confused. The digits 0-9 encoded in ASCII use 8 bits. Knoocking it down to 4 bits doesn't make your file 1/16th the size – Nick Keighley Nov 21 '18 at 14:43
1

Try looking at reading-binary-file-and-looping-over-each-byte. That should let you obtain the bytes.

Then you'll need to map each character you are interested in into a 4 bit representation. The binary operators are you're friend here, something like:

byteFirst = f.read(1);
byteSecond = f.read(1);

//assuming ascii/utf-8
encoded = ((byteFirst - 48) << 4) | ((byteSecond -48) << 4);

You'll need to wrap error handling logic around that, particularly for end of file, an odd number of characters, or any other character such as line ending, carriage return, space or other non-numeric character.

To decode the above:

byteFirst = ((encoded >> 4) + 48);
byteSecond = ((encoded & 15) + 48);

You'll need to make appropriate type casts too.

You may find it easier to use the built in compression schemes. You'll likely get better mileage in terms of size, and won't have to work directly with bits. On the other hand if you need record aligned storage, you may actually want a database engine instead.

The short of it is that bit manipulation is easy to get wrong, the compiler won't complain if you get the mask wrong, or the number of shifts, or even the size of the underlying bit-field. Its not that it is evil or should never be done - It is great for optimisation of speed/space - but does make understanding what is going on much harder.

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.