I am desiging a file format and I want to do it right. Since it is a binary format, the very first byte (or bytes) of the file should not form valid textual characters (just like in the PNG file header1). This allows tools that do not recognize the format to still see that its not a text file by looking at the first few bytes.

Any codepoint above 0x7F is invalid US-ASCII, so that's easy. But for Unicode it's a whole different story. Apart from valid Unicode characters there are private-use characters, noncharacters and sentinels, as I found in the Unicode Private-Use Characters, Noncharacters & Sentinels FAQ.

What would be a sentinel sequence of bytes that I can use at the start of the file that would result in invalid US-ASCII, UTF-8, UTF-16LE and UTF-16BE?

  • Obviously the first byte cannot have a value below 0x80 as that would be a valid US-ASCII (control)character, so 0x00 cannot be used.
  • Also, since private-use characters are valid Unicode characters, I can't use those codepoints either.
  • Since it must work with both little-endian and big-endian UTF-16, a noncharacter such as 0xFFFE is also not possible as its reverse 0xFEFF is a valid Unicode character.
  • The above mentioned FAQ suggests not using any of the noncharacters as that would still result in a valid Unicode sequence, so something like 0xFFFF is also out of the picture.

What would be the future-proof sentinel values that are left for me to use?

1) The PNG format has as its very first byte the non-ASCII 0x89 value, followed by the string PNG. A tool that read the first few bytes of a PNG may determine it is a binary file since it cannot interpret 0x89. A GIF file, on the other hand, starts directly with the valid and readable ASCII string GIF followed by three more valid ASCII characters. For GIF a tool might determine it is a readable text file. This is wrong and the idea of starting the file with a non-textural byte sequence came from Designing File Formats by Andy McFadden.

  • 3
    Since it is a binary format, the first bytes of the file should not form valid textual characters - You should look at the magic file (/usr/share/magic, or /etc/magic on many unix systems) that shows how this application identifies file types. A PNG file starts out with \x89PNG\x0d\0a\x1a\x0a -- note the "PNG" in there, that's a raw string. The sequences \x89 and the like are non-printable bytes.
    – user40980
    Mar 13, 2013 at 15:29
  • 7
    That was an example. A .gif starts out with GIF8. A SGI movi file starts out with MOVI. One style of zip archive file starts out with ZZ, the more popular pkzip format starts out with PK. The constraint that the first byte be an invalid text character does not seem to match what is found in the wild. I am curious why this is a requirement.
    – user40980
    Mar 13, 2013 at 15:51
  • 3
    Do you really care how other programs behave when they see a unknown file? To me, a signature sequence (like PNG files) is much more useful than a sentinel sequence - when the content is sent through a simple stream protocol, the receiver can immediately decide how to handle the following bytes. A Omani-sentinel sequence is next to no-sequence once everyone starts using it to identify their own format.
    – Codism
    Mar 13, 2013 at 16:51
  • 1
    @GrandmasterB What are the great benefits of not doing it? Is it in any way better? Is it so much effort to do it? In a new file format one can pick any magic bytes, so why not pick some that may have at least some benefits, no matter how small? Mar 13, 2013 at 19:17
  • 2
    @Virtlink, I dont particularly care what bytes you use in your file format. But you made an assertion that its 'wrong' to use ascii characters... yet I've not seen anything here that supports that claim, and there's plenty of empirical experience that shows it really doesn't matter (ie, the countless file formats that have been using ASCII characters without a problem for decades) Mar 13, 2013 at 19:49

3 Answers 3


0xDC 0xDC

  • Obviously invalid UTF-8 and ASCII
  • Unpaired trail surrogate in lead position regardless of endianess in UTF-16. It doesn't get more invalid UTF-16 than that.
  • But perfectly reasonable ISO-8859-1, and probably reasonable in any other character set that uses an 8-bit encoding.
    – parsifal
    Mar 13, 2013 at 23:06
  • 4
    +1 OP didn't ask for ISO 8859-1, just US-ASCII and UTF-*. Mar 14, 2013 at 0:00
  • @RossPatterson - true, but I suspect that's mostly because the OP hasn't really thought through the problem. Without any statistics to back me up, I'm willing to bet that a random "is this text" algorithm is more likely to give preference to ISO-8859-1 than UTF-16, simply because there's an enormous amount of 8-bit text in the world.
    – parsifal
    Mar 14, 2013 at 12:57
  • 3
    @parsifal Any binary is valid ISO-8859-1 so it doesn't need to be considered simply because it's impossible to make invalid ISO-8859-1.
    – Esailija
    Mar 14, 2013 at 13:05
  • 1
    @parsifal true and if that was the requirement you could just use 0x00 or whatever, but op didn't want that.
    – Esailija
    Mar 14, 2013 at 14:42
  • In UTF-8, the bytes C0, C1, and F5 - FF are illegal. The first byte must either be ASCII or a byte in the range C2-F4, any other starting byte is not valid UTF-8.

  • In UTF-16, the file normally starts with the Byte Order Mark (U+FEFF), otherwise applications have to guess at the byte order. Codepoints in the range D800-DBFF are lead bytes for a surrogate pair, and DC00-DFFF are the trailing bytes for a surrogate pair.

Thus, I'd use the byte combo F5DC. These two values are:

  • Not ASCII
  • Not valid UTF-8
  • Either interpreted as a UTF-16 trailing byte in a surrogate pair (not legal), or the codepoint U+F5DC, which is a private use character, but only by applications that stubbornly try to interpret this as UTF-16 even without a BOM.

If you need more options, F5DD through to F5DF all have the same 3 properties, as do F6DC - F6DF, F7DC - F7DF and F8DC - F8DF, for a total of 16 different byte combos to pick from.

  • So, by Esailija's suggestion to use U+DCDC, 0xDC would be valid UTF-8? Mar 13, 2013 at 16:38
  • 2
    @Virtlink 0xDC is a UTF-8 lead byte for a 2-byte sequence. It must be followed by a 10xxxxxx continuation byte for it to be valid. 0xDC is not a valid continuation byte, so 0xDC 0xDC is not valid UTF-8.
    – Esailija
    Mar 13, 2013 at 16:40
  • @Virtlink: No, because the second byte is not valid, it would have to be in the range 80 - BF. Mar 13, 2013 at 16:45

If you're trying to use a non-printable character to indicate "not text," then you'll find it hard to beat 0x89:

  • It's outside the US-ASCII range
  • In ISO-8859-1 it's a non-printable character ("CHARACTER TABULATION WITH JUSTIFICATION "). Likewise with Shift-JIS, which I believe is still in common use. Other 8-bit encodings may, however treat this as a valid character.
  • In UTF-8 it's an invalid first-byte for a multi-byte sequence (top bits are 10, which are reserved for characters 2..N of a multi-byte sequence)

Generally, when you form magic numbers, "non-text" is a minor point. I'll have to look up the reference, but one of the standard graphics formats (TIFF, I think) has something like six different pieces of useful information from its magic number.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.