If I was going to write a parser for HTTP, would I be able to assume the encoding of the HTTP headers and status line? Until I read the charset or encoding header, how could I tell what the encoding type was? I am given the impression that these lines will always be in ASCII.

I guess I am confused how HTTP handles various encodings within the same stream of data. I am getting the impression the status line and the headers can be in a different encoding than the body. Even in the case where the body is made up of multipart form-data, it sounds like the body has a single encoding. Some clarification/explanation would go a long way.

  • 4
    well, what does the HTTP specification say? If you're writing a parser, you'll probably be paying close attention to the grammar laid out in the spec. – amon Dec 6 '15 at 22:26
  • RFC 2616 at w3.org/Protocols/rfc2616/rfc2616.html has been superseded, but contains a link to the current RFCs that you need to examine carefully. – gnasher729 Dec 6 '15 at 22:28

RFC 7230, the relevant part of the current version of the spec, is pretty clear and to the point:

3. Message Format

A recipient MUST parse an HTTP message as a sequence of octets in an encoding that is a superset of US-ASCII*. Parsing an HTTP message as a stream of Unicode characters, without regard for the specific encoding, creates security vulnerabilities due to the varying ways that string processing libraries handle invalid multibyte character sequences that contain the octet LF (%x0A).

This allows for (at minimum) using a conformant UTF-8 parser, because UTF-8 avoids encoding confusing ASCII-subset characters in its multibyte code units, so e.g. %x0A will always be correctly recognized as an actual LF character.

There's a further note that, once you've successfully parsed the basic message into its header key-value pairs plus the message body, you can begin parsing the pieces with a more relaxed or non-default approach, according to certain headers. This is especially useful with RFC 7231's Content-Type header: Media Type

HTTP uses Internet media types [RFC2046] in the Content-Type (Section and Accept (Section 5.3.2) header fields in order to provide open and extensible data typing and type negotiation.

RFC2046 is all about extending MIME to message bodies, and has in turn a nice clear section on the Charset parameter:

4.1.2. Charset Parameter

A critical parameter that may be specified in the Content-Type field for "text/plain" data is the character set. This is specified with a "charset" parameter, as in:

Content-type: text/plain; charset=iso-8859-1

It goes on to explain that other text/ media types should use the same charset semantics.

Note that Content-Encoding, Transfer-Encoding, and Content-Transfer-Encoding (obsolete) all refer to a very limited set of encodings for compression or chunking — not character sets.

*American National Standards Institute, "Coded Character Set -- 7-bit American Standard Code for Information Interchange", ANSI X3.4, 1986.

  • Thanks. This really helps cut to the chase. The good news is the underlying platform (.NET in this case) is already handling this for me. The only question I have left is can the body be in a different encoding than the status line/headers. It sounds like in the case of multi-part the encoding has to match the rest of the body's, but it can provide a separate Content-Transfer-Encoding to indicate how to get back to the original encoding. – Travis Parks Dec 7 '15 at 2:57
  • @TravisParks: I think my edit should help explain all that. – Nathan Tuggy Dec 7 '15 at 4:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.