Create and implement new encoding

Question

I'm working on a project with huge files that contain only the set {[0-9],.}. Encoding in UTF-8 or ASCII make huge files.

I wonder if I could find a way to encode in only 4 bits (make those file 16 times smaller). I would have enough of 16 characters!

How could I proceed? I'm working on python and I have tried the following code. Seems like the computer doesn't want it, because it only erases what was in the text file...

def encodeData(self, filePath):
    print("File size before 4bits Encoding = {} Bits" .format(os.path.getsize(filePath)))
    print("Proceed with '{}' 4 bit Encoding... ".format(filePath))
    with open(filePath, "w+") as file2encode:
        for line in file2encode.readlines():
            for i in len(line):
                line[i] = 0x4
    print("File size After 4bits Encoding = {} Bits".format(os.path.getsize(filePath)))

Is there a way I can play with 4 bit encoding?

Answer 1

Try looking at reading-binary-file-and-looping-over-each-byte. That should let you obtain the bytes.

Then you'll need to map each character you are interested in into a 4 bit representation. The binary operators are you're friend here, something like:

byteFirst = f.read(1);
byteSecond = f.read(1);

//assuming ascii/utf-8
encoded = ((byteFirst - 48) << 4) | ((byteSecond -48) << 4);

You'll need to wrap error handling logic around that, particularly for end of file, an odd number of characters, or any other character such as line ending, carriage return, space or other non-numeric character.

To decode the above:

byteFirst = ((encoded >> 4) + 48);
byteSecond = ((encoded & 15) + 48);

You'll need to make appropriate type casts too.

You may find it easier to use the built in compression schemes. You'll likely get better mileage in terms of size, and won't have to work directly with bits. On the other hand if you need record aligned storage, you may actually want a database engine instead.

The short of it is that bit manipulation is easy to get wrong, the compiler won't complain if you get the mask wrong, or the number of shifts, or even the size of the underlying bit-field. Its not that it is evil or should never be done - It is great for optimisation of speed/space - but does make understanding what is going on much harder.