Best practices for reading dynamic byte streams: is line-by-line comparison with a mask the best way?

Question

I receive sensor data as a binary stream of bytes. This stream is not always the same length, and does not include the same data set each time. If the sensor did not send a field, it is simply absent, not null. The first 62 bytes of data is always predictable, but after that it's all up in the air. However, the message is broken up into sections, each with a header value that describes the proceeding data.

The old implementation was to read the header, and do a bitwise comparison with a bytemask to tell whether or not the field was populated. This occurred for each value that could have been written to the stream by the sensor. An example is as follows (C#):

    // declare bitmask
    private static int FIELD_SPECIFIC_BITMASK = 0x01;

    public Task Process(BinaryReader reader)
    {
         // read section's header
         int sectionHeader = reader.ReadInt32;

         if((sectionHeader & FIELD_SPECIFIC_BITMASK) !=0)
         {
              reader.ReadByte()
         }
    }

There has to be a more efficient and cleaner way of doing this write? There are dozens of fields across five sections, and this just seems inefficient, but I haven't been able to come up with a better solution.

Answer 1

When there are a limited number of field combinations that are sent relatively often, you can first do a switch on section header and jump to reading a canned sequence of fields. In the default branch of the switch, i.e. when reading a relatively rare combination of fields, you revert back to bit testing.

If your programming language supports it (c# does via System.Runtime.Interop.Marshal), you can use memmory mapped I/O instead of a bunch of BinaryReader.Read() operations. Marshal.PtrToStructure may be faster but it is limited to fixed structures. You have to measure whether it is faster in your situation. In addition the structures help documenting the binary protocol.