Okay, the question title may be a bit cryptic. But it aptly describes what I'm looking for:

I think every experienced coder went through this numerous times: You get a binary file format specification, you implement the reader for it, and… nothing works like expected. So you run your code in the debugger, go execute through the code line by line, every header field is read in seemingly correct, but when it comes to the bulk data, offset and indices no longer match up.

What would really help in this situation was a binary file viewer, that shows you the progress of your file pointer, as you step through the code, and ideally would also highlight all memory maps. Then you could see the context of the current I/O operations, most notably those darn "off-by-one" mistakes, which are even more annoying when reading a file.

Implementing such a debugger should not be too hard. traces on the process' file descriptors/handles and triggers on the I/O functions, to update the display.

Only: I don't know of such a kind of debugger to exist. Do I just lack knowledge about the existance of such a tool, or is there really no such thing?

closed as off-topic by gnat, jwenting, user40980, Bart van Ingen Schenau, pdr May 7 '14 at 15:25

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – gnat, jwenting, Community, Bart van Ingen Schenau, pdr
If this question can be reworded to fit the rules in the help center, please edit the question.

  • Was the decoder implemented with assertions and unit tests? Does it work correctly on smaller files and test samples? Do you have access to a "reference implementation"? – rwong Jul 4 '12 at 1:56
  • About assertions: Here is the first problem: There's no way to know if a value, you've read in is correct until you've read the whole file. And the only sanity check so far is, if after reading all the chunks in the file, the file pointer is exactly at the end of the file. If not, something went wrong. About the "reference implementation": The file itself is mostly a dump of a data acquisition program, done quick'n'dirty in LabView. – datenwolf Jul 4 '12 at 9:07
  • I have the VI this file was supposedly created with, but the pitfall here is, that its sooo easy to change the data layout of a binary dump without noticing. So the only way do pick this apart is reading the file, field by field, and also looking at the surroundings of the current file pointer position, if the context makes any sense. So technically I'm not writing the final reading code, but are first trying to nail down a working parser, which is then rewritten in the program propper. And no, a simple hexeditor doesn't suffice, because this a 6GiB of data of 1024 chunks with headers. – datenwolf Jul 4 '12 at 9:09
  • For the sake of fellow answerers, this is the LabView data format specification: zone.ni.com/reference/en-XX/help/371361H-01/lvconcepts/… – rwong Jul 6 '12 at 8:50
  • @rwong: After 2 sleepless nights I actually figured out the culprit. And like I suspected, it was a in-situ-change to the VI. In the data dump I got there's some additional double precision float in the per chunk headers, that was not included in file writing VI I was given. How I figured it: Well, I hacked up such a file I/O debugger myself, by having my code open the file in direct mode and having my I/O following hexviewer to keep in sync with /proc/$PID/fdinfo/$FD. Then I step executed my file reader in a execution debugger. – datenwolf Jul 6 '12 at 9:37

Suggestion #1: GUI tools:

Process Monitor is exactly such a tool available for free, on Microsoft Windows x86 and x64. http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

Suggestion #2: Build your own tools:

A complex file format can often be thought of as a deeply nested tree structure of "containers". The actual data on the file may be linked with offset pointers, but this doesn't prevent you from building a conceptual, hierarchical view of the file's structure.

What's the easiest way to display such a structure? XML.

Suggestion #3: implement your decoder to an interface that's debugger-friendly

Take a look at the IStream interface (COM), and consider refactoring your code to work with these elementary file operations:

  • Seek
  • Read
  • Write
  • SetSize

Along with a bunch of Decorator classes (implementing the IStream interface), you can implement readers for very sophisticated file formats.

As an example, the image encoders and decoders in Windows Imaging Component are implemented with such architecture.

This approach will not lead to great I/O performance; but if you can't get a correct implementation, focusing on performance will be futile.

  • Nice suggestion. But unfortunately my project is targeted for a Unix machine. There's neither ProcessMonitor nor IStream there. – datenwolf Jul 4 '12 at 0:43
  • I'm sure there are Linux equivalents. Feel free to edit / add examples to my answer. – rwong Jul 4 '12 at 0:44
  • Well, about your suggestion using XML: The file in question is about 6GiB in size. Basically it consists of a header of sampling parameters and then loads of sample dumps with the structure with the occasional descriptor inbetween. – datenwolf Jul 4 '12 at 0:47
  • My XML suggestion is only suitable for the higher-level overview of the file structure. – rwong Jul 4 '12 at 1:51

As every binary file format (Just like text files) is different, there cannot be a single tool that does it for everyone. Unless you write one yourself, or the writer of the spec has one for you, you will be out of luck. A decent editor that support binary viewing of files is mandatory, as a an ability and willingness to do Hex math.

Binary files are not hard, the only difference to a text file is that programmers every where seem to quake in their boots when the hear the term.

My experience (coming from embedded systems) is most people have problems because of bad assumptions - things like a Byte is 8 bits and a word is 32. A methodical approach and no assumptions are the best way to deal with your problems.

  • I'm not asking for a magic file analyzer. What I'm asking for is a debugger that keeps track of the open file descriptors, shows the file in a hex editor and displays the current file pointer position therein. Again: I'm not asking for a structural analyzer. Say you got a FILE*, a int fd, or a HFILE and instead of giving you the numeric value of the handle, you get a display of the file contents around the current file pointer position. – datenwolf Jul 4 '12 at 0:16
  • Also: I have no problems writing binary file writers or parsers. It's only getting problematic, if the only kind of spec you have is some C header file with a structure without packing and alignment pragmas and somebody just dumped this into a file. If you could see the context of your parser, and also make interactive adjustments to its progres this would ease up deciphering such binary a lot. – datenwolf Jul 4 '12 at 0:21
  • @datenwolf I kind of get what you want, but I think buffering will screw it up. For example, FILE* is a wrapper over file descriptors (in Unix). What if there are other wrappers on it? You'd need to take special care for each of them. – K.Steff Jul 4 '12 at 0:22
  • @K.Steff: Well, the debugger could offer a intelligent display mode for those wrappers. After all FILE* is part of the C stdlib and hence works the same for all binaries on a given system. So even if FILE is meant to be opaque, a debugger could be equipped with implementation specific knowledge. If a program uses a non-standard libc, then the debugger could bail out with a "unsupported libc" warning. – datenwolf Jul 4 '12 at 0:32
  • @datenwolf My point is if tomorrow you use another lib, (e.g.) optimized for big files, the debugger will have to know about its internals, too – K.Steff Jul 4 '12 at 0:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.