1

I'm about to create a specific file format for a particular project.

There is no special metadata related to the file, as it can be saved in the database, in a file, etc. So the extension of the file, the date, etc. don't matter at all: only the content does.

I now know that the format will have several iterations, so the first thing I'm doing is designing it with further versions in mind.

What I've thought is the following:

<format-id>       -> 4 bytes
<format-version>  -> 1 byte

Now the hard part: when I want to parse any version of the format, what should I do?

I've thought about using a PushbackInputStream, reading the first 5 bytes and fetch the appropriate version parser on the basis of those bytes, then unread the bytes and present the "pristine" stream to the actual parser. This way I can test parsers independently without using my "parser-aggregator".

Example:

byte[] identifier = "ABCD".getBytes(UTF_8);
try (PushbackInputStream in = ...) {
  byte[] header = IOUtils.readFully(in, new byte[5]);
  in.unread(header);
  if (!Arrays.equals(identifier, Arrays.copyOf(header, 4))) {
    throw new UnknownFormatException();
  }
  int version = header[4];
  getParser(version).parse(in);
}

The second way I've thought of is to present the start of stream to the parser and let it accept or refuse to parse the rest of the stream:

try (PushbackInputStream in = ...) {
  byte[] header = IOUtils.readFully(in, new byte[5]);
  in.unread(header);
  for (Parser parser: parsers) {
    if (parser.accept(header)) {
      parser.parse(in);
      break;
    }
  }
}

My question is: are there robust techniques to implement such a multi-format parser? For instance, how does Java Swing's ImageIO handle so many formats?

Notes:

  • I'm mostly trying to avoid each parser to try to parse the stream. I absolutely don't want that to happen. This question answers that, but that's not what I want.
  • Please discard any discussion about reusing an existing extensible format. That's out of scope and doesn't answer the question. – Olivier Grégoire Aug 11 '16 at 13:29
  • What's wrong with the solution you already proposed? – Robert Harvey Aug 11 '16 at 15:39
  • @RobertHarvey To me, nothing, so far. I made them myself. But I was wondering if there's something more generic for that use case, some kind of template. At least to get reassured I'm on the right track. I just never saw any example for this use case, despite actively looking for it. – Olivier Grégoire Aug 11 '16 at 18:25
2

Why bother re-parsing the header? By the time your version-specific parser has been instantiated, you already know what the header must have contained, so just parse the body.

If the header contains some other information the body parser needs, just pass it as an argument when you instantiate the body parser.

You can think of this as an encapsulated format [header][body], where the header tells you which body parser to use. If you need more metadata, the body can contain a nested [header2][body2], or whatever other format you need.

0

Create a stream. Pass it to a miniature parser that only looks at the format and version. Then re-create the stream and pass it to the correct parser.

This way, you don't instantiate any heavy-weight parsers until the need arises.

  • This is roughly what I wrote in my first example, except that I use a PushbackInputStream instead of opening the stream twice, isn't it? – Olivier Grégoire Aug 11 '16 at 14:04
  • I think I am describing a slightly different philosophy. There are some kind of streams that don't allow pushback, depending on your library. This means you have to research your library looking for limitations and caveats. Starting a new stream with the same content is essentially a fresh start. – BobDalgleish Aug 11 '16 at 14:54
  • 1
    Well, PushbackInputStream is part of Java since the first version of Java (question is tagged java, the code is Java-formatted, uses Java classes, etc.). Fresh start means re-looking for the file, re-opening it. Where I already have everything at that moment. Nevertheless, I todally understand the argument you explain about the heavy-weight parsers. That's definitely something I'll keep in my mind. Thanks! – Olivier Grégoire Aug 11 '16 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.