UPDATE 1 as requested by Brendan.
We are developing a Unix batch application for storing millions of customer records into a relational database. In order to allow multiple batch jobs to run in parallel, and to achieve a certain amount of concurrency while processing an input record, we've distributed the work across nine server daemons, all within the same LAN as the client, each of which is responsible for an isolated task (e.g. store the name in the name table, store the address in the address table, etc.).
Each daemon will be able to accept connections and requests (concurrently) from multiple clients.
Finally, whatever communication protocol we design (or adopt) must be extensible to accommodate different kinds of connections - for instance, we'd like for a monitor tool to be able to connect to a daemon, over the same port as the clients, and request statistics or send commands. There will also eventually be different kinds of clients that require different functions to be performed by the daemons. In other words, a client connecting to a daemon will have to be able to declare "I want to post customer records," or "I want to apply a batch of changes-of-address to the database" or "I want to send you a few commands."
I think it's a foregone conclusion that the underlying transport of these connections be TCP.
Finally, apart from the "command session", these will be multi-million record batch jobs, and performance is critical. The processing is iterative, and every record has the same layout (so wrapping each one in XML, say, would be unnecessary and highly wasteful).
I've no idea if this is enough information for someone to suggest some formats and protocols to use, but I'll gladly try to clarify or supply additional details if asked.
END OF UPDATE 1
(Below is the original post, which may or may not be of any value.)
I'm in charge of designing an application protocol for a set of in-house, batch-oriented, client-server applications. I'm familiar with IBM LU 6.2 (a.k.a. "Advanced Program-to-Program Communication"), and for the past 15 years have worked in Unix environments.
The client is a batch job that connects to a server process — which, in turn, connects to nine other "sub-server" processes. The client passes a customer transaction record to the server, which distributes different portions of the record to the sub-servers. Each sub-server processes its portion independently and concurrently, and passes some results back to the server which then passes them back to the client.
These are TCP connections, and we decided that, first of all, we'd use newline-terminated ASCII strings (we call these lines) as our Protocol Data Unit. A record is a line that is subdivided into tab-separated fields, and since each process will have its own data requirements, field#1 of every record will contain a record type. This is used as a key into a metadata store to find the field names, in the order they’re expected to appear in the record, in order to know how to interpret the record. For example:
Metadata: NAME=(prefix, first_name, middle_name, last_name, personal_suffix, professional_suffix)
(Here I use '|' to represent the tab character, and '#' to represent newline character)
Very straightforward so far, I think.
But then, new twists arose. The first was that, instead of sending all the data about one transaction in a single [large] record, we'd like to possibly use several records: a NAME record, an ADDRESS record, an EMAIL record, etc. So we invented the block - a series of records, preceded by a “special” BB record and followed by an EB record. The BB record’s first field is, of course, 'BB', and the second field holds the block type. Block type, like record type, is a key into metadata listing the record types (required, or optional) in this block type.
Understandably, each of these ten or so processes, having its own unique data requirements, has led to its own set of record types, enclosed in their own set of block types. That's a lot of metadata, but it isn't really a showstopper.
The next thing requirement that emerged was a special "context-setting" message, to transmit certain configuration variables that would remain constant from transaction to transaction. This gave rise to a new record and block type, as well as a new column in the block metadata that indicated this kind of block was somehow "special".
Then, envisioning circumstances where the "context" could change during the runtime of the client, we lifted the rule that stipulated there could be only one "context" message, at the beginning of a connection. At the same time, we realized there was still a need for a true "initialization" message that could only come once, at the beginning, conveying "session-global" parameters. Thus the INIT block was born, and now there were three flavors of block: session-initialization, context-switching, and normal application data.
Then, how about a connection that isn't for transaction-posting at all, but rather a control session for querying the health of the server, retrieving statistics, changing operational settings? The INIT block’s role became more general, declaring the "mode" of the session, which could now be either "posting", or "control."
We've been trying to wedge all of this into a single-layered application protocol, and it's getting out of hand. Just defining a "special" BB record should have raised a red flag - it's constructed from application-level objects and metadata, but it is not application data - rather, it frames the application data.
I think what has evolved is a 2-layer architecture atop TCP: one layer that simply provides typed containers (blocks and records), and a higher layer that uses those containers for a specific purpose, be it posting, or control, or something else - the true application protocols. However, I'm not well-versed enough in the design of protocols to recognize "patterns" that might be obvious to the pros. I'd appreciate any feedback on how to approach this.
It also seems possible that someone may have written a guide to designing protocols, that addresses these very concerns. I just haven't found it.
And my apologies for the length of this post. I hope none of it was immaterial. My deep thanks for your patience!