Restructuring a “processing slip” like object used in many steps

Question

I have worked on a library which processed data through multiple steps. It was written in R, a dynamic programming language where one could just add fields to existing “objects”, which were just glorified named lists (dictionaries). After loading from file, each object would contain a handful of members: two arrays with numbers, some meta data strings and integers. A typical analysis would then call pre-processing functions, which changes the numeric arrays and add more meta data fields indicating the transformations performed.

To give some examples: The starting point was called raw correlator. In some cases one would shift the time, and that modified data added a new field delta_t. But one could also weight and shift instead, which would modify data differently add delta_t and weight_factor to the meta data. Next one would bootstrap and that didn't change data, but added samples. The object could be passed into a plot() function at any time. Because we did not know which steps were done, introspection was used to figure out which field present. If samples was present, the error bars were derived from that, otherwise they were computed from data. Functions like add(correlator_1, correlator_2) would just add the data arrays element-wise. If there were samples, they got invalidated by removing the field from the result, and if delta_t was present but not the same, the function threw an error.

While working with the library I often had errors because some function expected a member which wasn't present in the correlator object that I have passed. By searching the code for the name of the member and finding a function which would set it, I learned which processing function to call first. It just felt wrong, but I couldn't really put my finger on it. The different states of the objects were not so different as to think of them as totally different objects. In any state one could for instance call plot() and it would give some sensible result. But thought in terms of strict C structs where all the members had to be listed at compile time, all these states were different types.

At the time I settled on grouping the members into some mixins and thought of the processing steps as adding more and more mixins. In other functions I could test whether a certain mixin was present instead of probing for certain members. These mixins were then called base correlator, shifted correlator, weight-and-shifted correlator, bootstrapped correlator and so on. Error messages got better, like “you need to bootstrap first”. It was an improvement, but it still didn't feel good.

The idea I had was to make them different types and nest them. So there would be a base correlator class where the fields were fixed. A next step, say bootstrap, would then have a different return value, the bootstrapped correlator class. That would have two members: base_correlator and samples. Functions which need a bootstrapped correlator can declare that with their type. All the old data would still be there, but as a sub-type. Adding two bootstrapped correlators is not possible, but one could just take their respective base correlator and bootstrap again. The types would support that.

However, there are optional steps like shift which do not affect other steps except in minor details. It doesn't affect plot(), but it did affect effmass() which would a different formula. Also nesting the types would mean that I needed to write an overload for plot() for both types. Having everything in one flat structure seemed easier and there was resistance to refactor to this nested structure.

Now I work on a system which handles requests and it appears to be the exact same situation. Now it is Python and the request class is also amended in various place in the code. Using introspection the presence of member fields is checked. The list of potential members doesn't fit on one screen page and makes me feel uneasy because there are so many states that objects could be in. I would like to add more structure to this.

The request comes in, the user has set a few text, flag and integer fields using the UI. The frontend also adds a few additional flags to the request. The middleware will then perform a few pre-processing steps depending on the values of contained flags. The modified request is handed to the backend part which handles the request and attached the result. The middleware does some more post-processing which adds even more fields. The UI eventually displays the result by picking out the result fields that it needs. All this is done within the same Python class.

I see the advantages in this: It is really easy to add another processing step in between, it might just modify existing fields. And if new metadata is required, it can just be added to the object anywhere. The original request fields can be looked at anywhere in the call chain. This approach has a lot of flexibility. And on the other hand I get the chills from all the implicit coupling. Since there are a lot of functions modifying a request object, they are all coupled with each other in non-trivial ways. Finding out what happens within a given function depends on which fields are present. Writing unit tests also feels hard because instances of that object need to be populated with at least the fields which are actually used in that function.

My idea of making these distinct classes like “user request”, “pre-processed request”, “handled request”, “post-processed request” and nesting the preceding one as a member wasn't liked so much. It would make it harder to work with the code because one would have to go up the nesting chain to retrieve of the fields from the initial user request. Also it would require changes in all functions until the end of the chain if another middle layer was to be introduced.

An alternative idea is to make the request class an aggregate of these types such that it would have the members user_request, pre_processed_request and so on. The functions would still change the request, but only add members to the step that they currently work on. They could easily read all the variables from previous steps. This grouping could scale if more groups had to be added in the middle.

Still all of this feels wrong. There is no encapsulation in either approach. And the sole desire to have the post-processing function look at the original user request appears like a violation of encapsulation. Yet the user might have ticked a checkbox which influences a post-processing step and therefore that function needs to look all the way back to the user input. I'd like to think of the steps in a functional way, having a strong type as input and output. On the other hand the request object serves as a processing slip (or docket, if that's a better word). In a factory there would be papers attached to a piece, each step would add more entries and each worker could look at the whole history of the piece.

Is there some standard pattern for this? Is my desire for encapsulation harmful in scenarios like this? How could one organize the data structure that threads through all the processing steps without it becoming a god object?

Answer 1

It sounds like the two approaches you've tried were:

• 1. Bundle everything into a single object that is modified all over the place
• 2. Pass old objects to new objects, so that they become hidden inside it

As you've noticed, both approaches will increase complexity over time - the former in a horizontal fashion, the latter in a vertical fashion.

What you need to do is to reduce the complexity as the data "flows" across your program. Instead of adding information to an object, you should try to relay information across objects.

Here's an example loosely inspired by your R experience:

class TimeShiftedCorrelator:
    def __init__(self, data, delta):
        self.data = data
        self.delta = delta


class WeightedTimeShiftedCorrelator:
    def __init__(self, data, delta, weight):
        self.data = data
        self.delta = delta
        self.weight = weight


class BootStrapCorrelator:
    def __init__(self, samples):
        self.samples = samples


class RawCorrelator:
    def __init__(self, raw_data):
        self.data = raw_data

    def shift_time(self, delta):
        shifted_data = external_lib.shift(data=self.data, delta=delta)
        return TimeShiftedCorrelator(data=shifted_data, delta=delta)

    def weighted_shift_time(self, delta, weight):
        weighted_shifted_data = external_lib.weight_shift(
            data=self.data,
            delta=delta,
            weight=weight,
        )
        return WeightedTimeShiftedCorrelator(
            data=weighted_shifted_data,
            delta=delta,
            weight=weight,
        )

    def bootstrap(self):
        samples = external_lib.bootstrap(data=self.data)
        return BootstrapCorrelator(samples=samples)


# User code example 
raw_correlator = RawCorrelator(raw_data='foo')
boot_correlator = raw_corr.bootstrap()

Now, when writing your code, you know what to expect from each RawCorrelator method because you know what kind of instance it returns. You don't need to wonder or inspect whether boot_correlator has the samples attribute or not, because any BootStrapCorrelator instance is guaranteed to have it.

At the same time, since we're not using inheritance or encapsulation, but rather just passing data across objects, every object keeps a lean body. It only has access to whatever it needs and nothing more. Does the object needs that data? Then give it to it as you construct it. BootStrapCorrelator knows nothing about the RawCorrelator that originated it, or its data, since (in my toy use case) it doesn't need it.

Each class could have its own plot method, instructing it how to be plotted. Or you could create a Plotter class with methods like plot_bootstrap_correlator, plot_raw_correlator, etc. In both cases, you won't need to wonder whether a required attribute is present, because you can simply assume that the object being plotted has all attributes expected for any member of its class.

Answer 2

In general, it's exceedingly rare for all the data to be needed in all the processing steps. There are probably exceptions, but I've personally never seen one.

What people do is break off the parts of the request they need and pass just those parts to the steps that process it. Then, they sort of assemble the parts into an output form.

It's sort of difficult to explain with your example without seeing it in detail, but to take an example my team is currently working on, one of our processing steps is to validate the uri.

The uri in string form is one field of a request object we receive, but we don't need the entire request just to validate the uri. We pull just the uri string from the request and parse and decode it into a form that looks like List("path", "to", "resource") instead of /path/to/resource. If the uri was syntactically invalid, we know it at this point, and can return an error to the user.

We then take the List("path", "to", "resource") form and pass it to an object that knows what our data model hierarchy looks like, so it knows if that resource actually exists. If it doesn't exist, we know it at this point, and can return an error to the user. Note that because the uri is passed to this code as a decoded list, we don't have to worry in this code about syntax and encoding problems with the uri.

Neither of these parts need to know anything about the headers of the request, or the body, or whether it's a GET or a POST, so we don't give them the entire request. They have access to exactly how much information they need to do their job. No more, and no less.

The other steps can be arranged similarly, where you use the request object when it first comes into the system, but the lower layers are using more cohesive objects.