C Equivalent of Perl Unpack & Pack

Question

I have some perl code that uses pack and unpack to transform the data in a way that I don't understand.

$unpacked_data = pack('b*', join('', unpack('(b7)*', $packed_data)))

Where $packed_data = a Byte Array of say, 2341 bytes, and with the above perl statement, unpacks to 2048 bytes of data.

Can someone explain this statement to me in terms of C? So far, I know this:

• b = A bit string (ascending bit order inside each byte, like vec()).
• If the repeat count is * , the offset is relative to the start of the packed string.

Answer 1

pack and unpack are functions within perl that can do some data transformations.

The innermost part of this is (b7)* - and you are correct that this is a bit string. However, it is only taking 7 bits at a time:

From: http://perldoc.perl.org/functions/pack.html

the b and B formats pack a string that's that many bits long. Each such format generates 1 bit of the result. These are typically followed by a repeat count like B8 or B64.

You are taking a string that is 8 bits long, chops off the high bit and rewriting it back out as 8 bit data chunks.

Starting from the beginning of the input string, each 8-tuple of characters is converted to 1 character of output. With format b , the first character of the 8-tuple determines the least-significant bit of a character; with format B , it determines the most-significant bit of a character.

If the length of the input string is not evenly divisible by 8, the remainder is packed as if the input string were padded by null characters at the end. Similarly during unpacking, "extra" bits are ignored.

For this, you would be taking something that is probably ASCII or similar 7 bit data that always has its high bit as 0, stripping off the leftmost bit (see below), and rewriting it out as a constant stream of data (rather than every 8th bit being 0).

The key to this is that it reads in 8 bits at a time with the b and only uses 7 of them.

---

Of note, there's a little bit of additional 'huh' going on in this code from how one typically looks at data and numbers. The b is reading the data little endian. This means that the leftmost bit is the one that is stripped off.

Lets take the string az and unpack it with b7. The result is 1000011 0101111 (I put a space there so that I could more easily read it and separate the bits).

'a' in binary is 01100001 and 'z' is 01111010 as big indian. Note the leftmost bit is 0.

unpack('(b7)*','az') will read it little endian (backwards from what one typically think) and drop the least significant (little endian) bit which is the leftmost one.

Now, that the always 0 leftmost bit of some (probably) ascii is gone, the entire thing is rewritten back out.

0110000101111010
^!      ^!

becomes

11000011111010
!      !

(the ^ and ! show equivalent spots between the lines)

Answer 2

I believe that you have the sizes of the strings backwards.

If we start with 2048 bytes, that's 16384 bits. Which is then broken up into 2341 groups of 7, each of which is put into a byte with the top bit 0. (And the trailing bits in the last one are padded with 0.)

One would presume that the operation actually began with someone's idea for how to (poorly) compress pure ASCII text - namely since the top bit in each character is always 0, just store the first 7 bits. This operation seeks to reverse that compression. Which means that this will lose data when it runs across any data with the top bit set - for example if there is UTF8 unicode in the input.

If you know the original author of that code, this might make a good teaching moment. Explain why data loss is not acceptable, show that person https://code.google.com/p/snappy/, and make the world a better place when crappy code doesn't break unexpectedly.