Selecting the endianness of data in files generated by an embedded system

Question

I have an embedded system running on a little-endian Cortex-M3. This system is able to accept packets from the network and reply to them. Now, I would like the system to start generating files. These files will then be downloaded and inspected with tools running on regular x86 desktop computers (i.e. also little endian).

For network packets, the best practice is to use big-endian (i.e. network byte order) for packet structures. I designed my packets to always use big-endian for network packets.

Now, I would like to have my system read from files and generate files, all with custom formats/structures. Is there a widely accepted endianness when it comes to files? Why would you select one endianness over the other, particularly when working with embedded systems? I am tempted to keep my files at little-endian, as this will simplify the code (I can just write structures directly from memory), but this makes it inconsistent with network data interface structures that I'm using. Is this considered acceptable?

Also, if these files will contain copies of big endian data (e.g. the network packets I mentioned), and the files are designed to be little-endian, is it a good practice to have the embedded system convert this big endian data to little endian before storing? Shouldn't the file be kept consistent throughout, or does it really not matter?

Note: as this is an embedded system, and bandwidth/storage/compute resources are limited, text formats are not a good idea in general in this case, so I couldn't avoid this. To reduce complexity, serialized formats are also not a good idea in my case.

Answer 1

If your goal isn't interoperability with independently written software, endianness conversion is probably overkill. You're already doing it for network data, which would not strictly be necessary - the "network endianness" is important for packet headers so that they can be handled by the network infrastructure such as routers etc., but application-level payload does not need to do this.

If you can do a synchronized incompatible upgrade of network data format, embedded software and desktop software, I would ditch the endianness conversion and simply declare that little-endian is what is used by your devices and tools.

A different but somewhat more open approach (still incompatible) would be to use protocol buffers as a language and framework neutral binary interchange format. You need to decide whether you need the flexibility and are willing to pay the complexity price.

But if you need to be backward compatible, you've painted yourself into a corner. Whichever way you decide will cause some pain, you get to decide which you consider least painful. There's no standard solution. Keeping file-exclusive data little endian makes generating and processing easier, keeping network packet data verbatim in files avoids additional conversion steps with associated risks, having separate endianness for different data in the file makes the format gross.