1

I am member of the Apache PLC4X (incubating) project. Here we are currently implementing multiple industry PLC protocols. While we initially focussed on creating Java versions of these, we are currently starting to work on also providing C++ and other languages.

Instead of manually syncing and maintaining these, we would rather define the message structures of these protocols in a generic way and have the model, parsers and serializers generated from these definitions.

I have looked at several options: 1) Protobuf 2) Thrift 3) DFDL

The problems with these are the following:

1) Protobuf seems to be ideal do design a model and have model, serializers and parsers generated from that. With Protobuf it is easy to define a model and ensure I can serialize an object and deserialize it with any language. However I don't have full control over the transport format. For example if I was to encode the constant byte value of 0xFF, this would be a problem.

2) Thrift seems to be more focussed on the services and the models used by these services. The same limitations seem to apply as for Protobuf: I have no full control over the transport format

3) DFDL seems to be exactly what I'm looking for as I want a language to describe my data-format ... unfortunately I could find projects like Daffodil, which seem to be able to use DFDL definitions to parse any data format into some XML like Dom structure. For performance and memory reasons we would rather not do that. Other than that I couldn't find any usable tooling.

Also had a look at Avro and Kaitai Struct but Avro seems to have the same issues for my usecase as Protobuf and the guys from Kaitai told me serialization was still experimental

My ideal workflow would be (Using Maven):

1) For every protocol I define the DFDL documents describing the different types of messages for a given protocol

2) I define multiple protocol implementation modules (one for each language)

3) I use a maven plugin in each of these to generate the code for that particular language from those central DFDL definitions

Improve this question
4
  • One practical alternative to model-driven-development/code generation might be to specify a shared test suite for all implementations, and then develop the implementations independently. This test suite could contain examples of messages that must be generated or parsed, and language-agnostic descriptions of operations your API must provide. This approach gives you strong guarantees and full control, at a cost of increased implementation effort.
    – amon
    Jan 9, 2019 at 16:42
  • Could you explain in more detail the issues you have with the transport format for protobuf and Thrift?
    – BobDalgleish
    Jan 9, 2019 at 17:15
  • @BobDalgleish Well to me it seems as if the binary output format is a result of the used model and you can use it nicely to serialize and deserialize a given model. However in my case we have the opposite situation. We have a given binary protocol, which we can't change. Here we have things like certain byte values representing different enum values and certain byte values being interpreted as indicators for different data structures. To me it looked as if it was way more work to get the tools to consume and produce a given output format than to implement things by hand.
    – Christofer Dutz
    Jan 10, 2019 at 8:48
  • @amon That would have been our current plan, however implementing the (de)serialization and model types for every protocol (we're talking of about 10-20 of them) is a very manual task especially if we want to support multiple languages (Java / C++ is not where it should stop, we plan on much more). So with a formal definition and code generator we could easily support all protocols on a new language by whipping up a new generator. Also could we generate the documentation (like here: plc4x.apache.org/protocols/s7/s7comm.html) and all languages would be perfectly in sync.
    – Christofer Dutz
    Jan 10, 2019 at 8:52

2 Answers 2

Reset to default
0

According to IBM

Data Format Description Language (DFDL) 1.0 is a modeling language from the Open Grid Forum that is used to define the structure of general text and binary formatted data in a way that is independent of the data format.

The particular problem that I see is the fact, that at least with Thrift (and proto) the data format is partially in the code generated from the IDL. For efficiency purposes, Thrift serializes all fields using numeric field IDs, not via their names. The field ID can (should) of course be specified in the IDL, but the number alone wont tell you much about the fields contents and intention.

If that, however, can be modeled via DFDL (can it?) then why don't you write a generator to generate Thrift IDL from your DFDL documents?

Same strategy could be applied to proto files, Avro, XML schema ... you name it. That way you have one source and everything else is generated.

Improve this answer
1
  • Well that's what I'm actually looking into. Being an Apache project, I'm currently working with the guys from the Daffodil project. It seems that I would be able to model the data formats with DFDL. Daffodil would be able to process these. What I would probably have to do, is to get my hands dirty and participate in the Project by writing the code generator. But in that case I probably wouldn't make it generate thrift stuff from that, but use project specific templates for writing code that integrates into our Netty code.
    – Christofer Dutz
    Jan 11, 2019 at 14:16
0

I can not at the moment investigate if this will fully meet your criteria but take a look at Cap'n Proto which at first look is more up your alley then Protobuf. Although you have much more control over the encoded data it doesn't seem to be full control still.

Improve this answer
3
  • Well we didn't find a perfect solution for our case. DFDL was indeed quite well suited for modeling the parsing and serializing, however the speed of interpreting everything was about a factor of 50-60 slower. So we concentrated on generating static code from the DFDL spec, but hence lacking inheritance the generated output was not very elegant and there were some internal dislikes of the complicated format (Guess cause I grew up with all the XML goodness from the 2000s and later, I had no problems with, but the other project members weren't overwhelmed). So we ended up with our own format.
    – Christofer Dutz
    Jun 11, 2019 at 19:17
  • With this format the others agreed that it is super-simple to understand and the generated code is pretty beautiful ... and the generated code was even 3 times faster than the already pretty fast one I initially developed.
    – Christofer Dutz
    Jun 11, 2019 at 19:18
  • @ChristoferDutz good for you on having the manpower :) and I kind of suspected you wouldn't use Cap'n since it doesn't give full control over wire format. It's more like C structs if anything.
    – jaskij
    Jun 11, 2019 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.