13

When programming in C I have found it invaluable to pack structs using GCCs __attribute__((__packed__)) attribute so I can easily convert a structured chunk of volatile memory to an array of bytes to be transmitted over a bus, saved to storage or applied to a block of registers. Packed structs guarantee that when treated as an array of bytes it will not contain any padding, which is both wasteful, a possible security risk and possibly incompatible when with interfacing hardware.

Is there no standard for packing structs that works in all C compilers? If not then I am an outlier in thinking this is a critical feature for systems programming? Did early users of the C language not find a need for packing structs or is there some kind of alternative?

  • using structs across compile domains is a very bad idea, in particular to point at hardware (which is another compile domain). packet structs are only one trick for doing this, they have a lot of bad side effects, so there are many other solutions to your problems with less side effects, and that are more portable. – old_timer Jan 15 '16 at 2:46
12

In a struct, what matters is the offset of each member from the address of each struct instance. Not so much is the matter of how tightly things are packed.

An array, however, matters in how it is "packed". The rule in C is that each array element is exactly N bytes from the previous, where N is the number of bytes used to store that type.

But with a struct, there is no such need for uniformity.

Here's one example of a weird packing scheme:

Freescale (who make automotive microcontrollers) make a micro that has a Time Processing Unit co-processor (google for eTPU or TPU). It has two native data sizes, 8 bits and 24 bits, and only deals with integers.

This struct:

struct a
{
  U24 elementA;
  U24 elementB;
};

will see each U24 stored its own 32 bit block, but only in the highest address area.

This:

struct b
{
  U24 elementA;
  U24 elementB;
  U8  elementC;
};

will have two U24s stored in adjacent 32 bit blocks, and the U8 will be stored in the "hole" in front of the first U24, elementA.

But you can tell the compiler to pack everything into its own 32 bit block, if you want; it's more expensive on RAM but uses less instructions for accesses.

"packing" doesn't mean "pack tightly" - it just means some scheme for arranging elements of a struct wrt the offset.

There is no generic scheme, it is compiler+architecture dependent.

  • 1
    If the compiler for the TPU rearranges struct b to move elementC before any of the other elements, then it is not a conforming C compiler. Element rearrangement is not allowed in C – Bart van Ingen Schenau Jan 13 '16 at 14:33
  • Interesting but U24 is not a standard C type en.m.wikipedia.org/wiki/C_data_types so it is not surprising that the complier is forced to handle it in a somewhat odd way. – satur9nine Jan 14 '16 at 16:45
  • It shares RAM with the main CPU core which has a word size of 32 bits. But this processor has an ALU which only deals with 24 bits or 8 bits. So it has a scheme for laying out 24 bit numbers in 32 bit words. Non-standard, but a great example of packing and alignment. Agreed, it is very non-standard. – RichColours Jan 15 '16 at 19:14
6

When programming in C I have found it invaluable to pack structs using GCCs __attribute__((__packed__)) [...]

Since you mention __attribute__((__packed__)), I assume your intention is to eliminate all padding within a struct (make each member have a 1-byte alignment).

Is there no standard for packing structs that works in all C compilers?

... and the answer is "no". Padding and data alignment relative to a struct (and contiguous arrays of structs in stack or heap) exist for an important reason. On many machines, unaligned memory access can lead to a potentially significant performance penalty (though becoming less on some newer hardware). In some rare-case scenarios, misaligned memory access leads to a bus error that is unrecoverable (may even crash the entire operating system).

Since the C standard is focused on portability, it makes little sense to have a standard way to eliminate all padding in a structure and just allow arbitrary fields to be misaligned, since to do so would potentially risk making C code non-portable.

The safest and most portable way to output such data to an external source in a way that eliminates all padding is to serialize to/from byte streams instead of just trying to send over the raw memory contents of your structs. That also prevents your program from suffering performance penalties outside of this serialization context, and will also allow you to freely add new fields to a struct without throwing off and glitching the entire software. It'll also give you some room to tackle endianness and things like that if that ever becomes a concern.

There is one way to eliminate all padding without reaching for compiler-specific directives, though it's only applicable if the relative order between fields does not matter. Given something like this:

struct Foo
{
    double x;  // assume 8-byte alignment
    char y;    // assume 1-byte alignment
               // 7 bytes of padding for first field
};

... we need the padding for aligned memory access relative to the address of the structure containing these fields, like so:

0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
x_______y.......x_______y.......x_______y.......x_______y.......

... where . indicates padding. Every x must align to an 8-byte boundary for performance (and sometimes even correct behavior).

You can eliminate the padding in a portable way by using an SoA (structure of array) representation like so (let's assume we need 8 Foo instances):

struct Foos
{
   double x[8];
   char y[8];
};

We've effectively demolished the structure. In this case, the memory representation becomes like this:

0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
x_______x_______x_______x_______x_______x_______x_______x_______

... and this:

01234567
yyyyyyyy

... no more padding overhead, and without involving misaligned memory access since we're no longer accessing these data fields as an offset of a structure address, but instead as an offset of a base address for what is effectively an array.

This also carries the bonus of being faster for sequential access as a result of both less data to consume (no more irrelevant padding in the mix to slow down the machine's relevant data consumption rate) and also a potential for the compiler to vectorize processing very trivially.

The downside is that it's a PITA to code. It's also potentially less efficient for random access with the larger stride in between fields, where often AoS or AoSoA reps will do better. But that's one standard way to eliminate padding and pack things as tightly as possible without screwing with the alignment of everything.

  • 2
    I'd argue that having a means of specifying structure layout explicitly would massively enhance portability. While some layouts would lead to very efficient code on some machines and very inefficient code on others, the code would work on all machines and would be efficient on at least some. By contrast, in the absence of such a feature, the only way to make code work on all machines is likely to be to either make it inefficient on all machines or else use a bunch of macros and conditional compilation to combine a fast non-portable program and a slow portable one in the same source. – supercat May 25 '16 at 15:29
  • Conceptually yes, if we could specify everything down to bits and bytes representations, alignment requirements, endianness, etc and have a feature that allows such explicit control in C while optionally divorcing it further from the underlying architecture... But I was just talking about ATM -- currently the most portable solution for a serializer is to write it in a way such that it doesn't depend on the exact bits and bytes representations and alignment of data types. Unfortunately we lack the means ATM to do otherwise effectively (in C). – user204677 Dec 31 '17 at 8:42
5

No all architectures are the same, just turn on the 32 bit option on one module, and see what happens when using the same source code and the same compiler. Byte order is another well known limitation. Throw in floating point representation and the problems get worse. Using Packing to send binary data is non-portable. To standardize it so it was practically usable, you would need to redefine the C Language specification.

Although common, using Pack to send binary data is a bad idea if you want security, portability or longevity of the data. How often do you read a binary blob from a source into your program. How often do you check all the values are sane, that a hacker or program change has not 'got' to the data? By the time you have coded a check routine, you might as well be using import and export routines.

0

A very common alternative is "named padding" :

struct s {
  short s1;
  char  c2;
  char  reserved; // Padding
};

This does assume the structure will not be padded to 8 bytes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.