Application protocols and state machines: designing a complicated server

Question

I'm designing and prototyping a server that has the following characteristics:

• during the daemon's initialization it spawns nine "manager" servers, each of which solves one part of the overall problem.
• after initialization, the daemon accepts connections from clients, forking a "slave" server for each which connects to the managers and then accepts requests from the client, farms them out to the managers, assembles the results, and sends them back to the client, ad infinitum.
• the daemon, the managers, and the slaves all should have the ability to concurrently accept "control" connections from a terminal, and receive and process commands (e.g. queries for performance stats).
• the concurrency of each process's socket I/O is being managed by select().

The problem I'm having is the complexity of the coding. For one thing, each program has at least two phases - initialization and main loop - in which it is variously establishing a connection to its parent, or accepting a connections from its children, or sending and asynchronously awaiting data from its children, and so on. At various points I don't want, say, to accept a client connection while I'm initializing, or I do want to accept control sessions even in the midst of regular processing. Not to mention managing the protocol of each such connection.

Given enough time, I could probably massage all of this into a reasonably well-factored set of subroutine calls. But I remember having heard of an entirely different approach to this sort of problem, involving libevent, callbacks, and finite state machines.

What I'm hoping is that someone reading this might say "Why, yes! That's a perfect example of where event callbacks and FSMs are far superior to looping on select()s - and a great introductory article about this is ..." or "here's what it's all about in a nutshell." (Or, possibly, "that's not what libevent is intended for, at all.")

Answer 1

There are no silver bullets in design or architecture. ( Sorry )

It sounds like you're still at the beginnings of the Design phase and I think you need to focus more on the "divide and conquer" aspects of mapping out your application. Rushing into the lower level details before you have a comprehensive design means you may miss otherwise "obvious" opportunities for particular techniques. Comprehensive doesn't mean final it just means that you've got all of the major components and their interactions laid out in a general form. In other words, you've got to know where you're trying to go before you can get there.

Once you've mapped out the major requirements of your daemon, managers, and processors (slaves) then you should start examining what requirements / functions are allowed at what stages. This will lead into your state definitions, and those states should help refine the scope of the program.

You should also be able to identify the common components based upon the functions and requirements. Combined with the state definitions from before, you can determine when common components can be called and with what parameters. Again, this should help scope down the size of your program.

It's at this point that you can consider queues, callbacks, etc... based upon what your design calls for the program to be doing. Prototype some and then come back and refine the design.