This is a complex problem, so to be clear the issues that need to be solved are:
- Display - How should data be rendered so as to appear sensible to a human
- Intent - How should the platform behave around the data
- Response - What should the platform collect, and what should the platform do with it.
- Integration - How to manage the interests of several sources of configuration.
- Configuration - How do you express all the necessary data to orchestrate your application.
- Communication and Verification - How do you communicate this to humans, and allow those humans to be sure that they got it right.
You probably have solved some of these. I'm detailing points 1, 2, and 3 to setup my argument for points 4, 5 and 6.
This is a solved problem: MVC and any of its variations will do.
A useful variation runs roughly:
- The UI is composed of Models, Views, and Controllers
- Models can be composed of models.
- Views can be composed of views.
- Controllers can be composed of controllers.
- A view renders one model, and has one slot for a controller per kind of interaction available with that view.
- a model can be rendered simultaneously or serially by many views, and has one slot for a controller per kind of state change.
- A controller acts on models and views in response to an interaction with a view, or a model changing state. The controller takes a model describing salient facts about the event, and transforms it to perform its interaction.
Define a small set of models for representing specific types of data.
- Boolean, Integer, Text
- Lists of X
- Graphs of X vs Y
As many or as few that form primitives for your display. Where possible define models in terms of more primitive models.
Define a set of views for rendering data to the display device. A good rule of thumb is to have a one-to-one with each model, but it is perfectly reasonable for a model to not have a view, or for there to be many views for that model.
Where possible make views delegate responsibility for rendering submodels, to subviews. But this does not always make sense, like displaying a graph in tabular (each cell using the integer subview) vs pictorial form (the list of integers for a series using a line subview).
There are three types of controller:
- Primitive controllers that perform one single well defined state-change on a model, or view.
- Composite Controllers that compose other controllers to form some sort of sequential, controlled repetition, or selective behaviour.
- A Business Controller that passes a copy of a model to the business logic.
These should be defined clearly, and simply.
The models passed to the controller should be the necessary and sufficient description of the interaction or state change such that no one needs to know who generated the event in order to obtain further details. For any particular case this might be empty, one or two values, right up to a copy of the the entire state of the view and/or model.
Interacting with Business Logic
From the perspective of the UI, the Business logic is just another user - it just doesn't need a picture drawn for it. That being said it still needs a view, or views. This provides a model that can be read, and a set of actions that the business logic can perform.
Configuration and Schemas
The configuration for the UI will hold:
- view configuration (font size, alignment, which sub-views are composed, ...)
- model configuration (the specific data to be displayed, captured)
- controller configuration (to link up how interaction with a view changes state in other views, and/or models)
- a schematic for linking views, models, and controllers together into a whole.
To be clear the configuration can supply many useful defaults, or a process to obtain a nuanced default, or user originated configuration. It may even make use of templates detailing common setups of model/view/controller. Placing all the burden of UI specification at the upstream services feet, is just moving the problem upstream and turning your platform into a fancy UI library/web browser.
The minimum configuration needed would be just the configuration for the models.
This is also a solved problem: Its called business logic.
This might live on the client, the server, or be somehow divided or shared. However the layer is organised, there will be processes that are responsible for modeling or orchestrating certain behaviours. Expose these processes to be configured by a configuration object.
Be careful in what you allow to be configured from your business layer, it represents a sizeable security risk as it is a form of unrestricted
eval() using other peoples code in your application. The configuration object is a form of code.
This allows the application to behave fundamentally different every day based on the data received. For example: collected results may be forwarded to another service, or it may send a report to the UI's user.
This is a harmonious relationship between the Rendering and the business behaviour.
The idea is that some of the UI models can be configured to allow the Human to input a response through clicking, key presses, voice, etc.. which populate the views respective model.
Then the business layer receives the model and proceeds to validate, and act on the data in that model as the configuration requires.
Of course if you only want a simple response, a direct model to text mapping, and sending via a file, or upload. If you want more complex interactions, and more configurable interactions, then the configuration of the UI and Business Logic Layer will become more complex.
The devil is often in the details. The application is serving as a Platform for two or more upstream definitions of behaviour. To get anywhere you will need to address how they will co-operate.
The no fuss answer is not at all. Each upstream service is dealt with independently. The user may be able to switch their views between each like tabs in a browser, but each functions independently of the others.
On the other extreme is the route of web-pages. The page is broken down into sub-pages each being sourced from potentially different sources. Each sub-page holds its own models, views, controllers, and schematic for assembling that part. They are even capable of communicating with each other by affecting views (some rendered other not) exposed by other parts of the overall page. This can become a mess as there are a number of actions that can be taking by one sub-page that are detrimental to other sub-page or to the overall page.
In the middle is a hard ground. Here the competing interests of the services are constrained by well defined collaboration interfaces, the more of these interfaces supported the more capable the platform, and the more likely that bad-actors can ruin the platform for everyone.
The shape of the configuration objects that are consumed by the views, models, controllers, display schema, and business process is your schema. There is nothing else that your platform can understand.
The main problem seems to be a lack of understanding of your platforms schema. Which is reasonable, any large application has a mind boggling schema.
My suggestion is not to worry about self-describing data. What you want to know is: from a given service, what is the top level configuration object?
This piece of information alone completely describes how that service will interact with your platform. If all of the upstream services provide the same object then this becomes really simple, otherwise there will be several high-level entry-points.
The top-level configuration object configures a Business Process. From here the process is responsible for unpacking that top object, which includes defining the data shape. Each sub-configuration object is passed off to another business process. Each responsible for unpacking the top object, defining what makes that object valid and distributing sub-configurations.
Each process that is interpreting the meaning of a configuration object is responsible for assembling the results of each sub-configuration, along with any warning, or error diagnostics and passing it back without side-effects. These processes should not cause side-effects because it undermines the utility of process. By having no side-effects while comprehending, errors can be handled without messy cleanup procedures, the state of the comprehended configuration can be inspected by tests, and it allows for a clean modularisation. Ideally the exact types being configured would be hidden through an injected factory.
If side-effects must happen before the configured state returned is usable, add an
activate() or similar method to the result. This allows the process needing a configured output to decide if it needs those side-effects or if its just interested in validating.
Communication and Verification
Now you could try and find a third-party data definition language, or force the configuration to carry useless validation data that your program does not care about.
Or because the business process for comprehending the configuration was constructed modularly, this can be placed in a library. That library plus a factory that constructs simple data objects instead of the complex platform objects, can be placed in a simple application that verifies text files, etc... for the expected configuration format, and display diagnostics should an error be encountered.
You will still need to communicate what the configuration format is between humans. But this can be done at a much more human friendly level in a human friendly language. And the humans can submit their mocked up data, or system output to the custom validator and get a real diagnostic back, about how the real system will interpret it, because the real comprehension (and validation) is being used.