I have worked on a library which processed data through multiple steps. It was written in R, a dynamic programming language where one could just add fields to existing “objects”, which were just glorified named lists (dictionaries). After loading from file, each object would contain a handful of members: two arrays with numbers, some meta data strings and integers. A typical analysis would then call pre-processing functions, which changes the numeric arrays and add more meta data fields indicating the transformations performed.
To give some examples: The starting point was called raw correlator. In some cases one would shift the time, and that modified
data added a new field
delta_t. But one could also weight and shift instead, which would modify
data differently add
weight_factor to the meta data. Next one would bootstrap and that didn't change
data, but added
samples. The object could be passed into a
plot() function at any time. Because we did not know which steps were done, introspection was used to figure out which field present. If
samples was present, the error bars were derived from that, otherwise they were computed from
data. Functions like
add(correlator_1, correlator_2) would just add the
data arrays element-wise. If there were
samples, they got invalidated by removing the field from the result, and if
delta_t was present but not the same, the function threw an error.
While working with the library I often had errors because some function expected a member which wasn't present in the correlator object that I have passed. By searching the code for the name of the member and finding a function which would set it, I learned which processing function to call first. It just felt wrong, but I couldn't really put my finger on it. The different states of the objects were not so different as to think of them as totally different objects. In any state one could for instance call
plot() and it would give some sensible result. But thought in terms of strict C structs where all the members had to be listed at compile time, all these states were different types.
At the time I settled on grouping the members into some mixins and thought of the processing steps as adding more and more mixins. In other functions I could test whether a certain mixin was present instead of probing for certain members. These mixins were then called base correlator, shifted correlator, weight-and-shifted correlator, bootstrapped correlator and so on. Error messages got better, like “you need to bootstrap first”. It was an improvement, but it still didn't feel good.
The idea I had was to make them different types and nest them. So there would be a base correlator class where the fields were fixed. A next step, say bootstrap, would then have a different return value, the bootstrapped correlator class. That would have two members:
samples. Functions which need a bootstrapped correlator can declare that with their type. All the old data would still be there, but as a sub-type. Adding two bootstrapped correlators is not possible, but one could just take their respective base correlator and bootstrap again. The types would support that.
However, there are optional steps like shift which do not affect other steps except in minor details. It doesn't affect
plot(), but it did affect
effmass() which would a different formula. Also nesting the types would mean that I needed to write an overload for
plot() for both types. Having everything in one flat structure seemed easier and there was resistance to refactor to this nested structure.
Now I work on a system which handles requests and it appears to be the exact same situation. Now it is Python and the request class is also amended in various place in the code. Using introspection the presence of member fields is checked. The list of potential members doesn't fit on one screen page and makes me feel uneasy because there are so many states that objects could be in. I would like to add more structure to this.
The request comes in, the user has set a few text, flag and integer fields using the UI. The frontend also adds a few additional flags to the request. The middleware will then perform a few pre-processing steps depending on the values of contained flags. The modified request is handed to the backend part which handles the request and attached the result. The middleware does some more post-processing which adds even more fields. The UI eventually displays the result by picking out the result fields that it needs. All this is done within the same Python class.
I see the advantages in this: It is really easy to add another processing step in between, it might just modify existing fields. And if new metadata is required, it can just be added to the object anywhere. The original request fields can be looked at anywhere in the call chain. This approach has a lot of flexibility. And on the other hand I get the chills from all the implicit coupling. Since there are a lot of functions modifying a request object, they are all coupled with each other in non-trivial ways. Finding out what happens within a given function depends on which fields are present. Writing unit tests also feels hard because instances of that object need to be populated with at least the fields which are actually used in that function.
My idea of making these distinct classes like “user request”, “pre-processed request”, “handled request”, “post-processed request” and nesting the preceding one as a member wasn't liked so much. It would make it harder to work with the code because one would have to go up the nesting chain to retrieve of the fields from the initial user request. Also it would require changes in all functions until the end of the chain if another middle layer was to be introduced.
An alternative idea is to make the request class an aggregate of these types such that it would have the members
pre_processed_request and so on. The functions would still change the request, but only add members to the step that they currently work on. They could easily read all the variables from previous steps. This grouping could scale if more groups had to be added in the middle.
Still all of this feels wrong. There is no encapsulation in either approach. And the sole desire to have the post-processing function look at the original user request appears like a violation of encapsulation. Yet the user might have ticked a checkbox which influences a post-processing step and therefore that function needs to look all the way back to the user input. I'd like to think of the steps in a functional way, having a strong type as input and output. On the other hand the request object serves as a processing slip (or docket, if that's a better word). In a factory there would be papers attached to a piece, each step would add more entries and each worker could look at the whole history of the piece.
Is there some standard pattern for this? Is my desire for encapsulation harmful in scenarios like this? How could one organize the data structure that threads through all the processing steps without it becoming a god object?