Data Transformation Pipeline

Question

I have create some kind of data pipeline to transform coordinate data into more useful information.

Here is the shell of pipeline:

public class PositionPipeline
{
    protected List<IPipelineComponent> components;

    public PositionPipeline()
    {
        components = new List<IPipelineComponent>();
    }

    public PositionPipelineEntity Process(Position position)
    {
        foreach (var component in components)
        {
            position = component.Execute(position);
        }

        return position;
    }

    public PositionPipeline RegisterComponent(IPipelineComponent component)
    {
        components.Add(component);
        return this;
    }
}

Every IPipelineComponent accepts and returns the same type - a PositionPipelineEntity. Code:

public interface IPipelineComponent
{
    PositionPipelineEntity Execute(PositionPipelineEntity position);
}

The PositionPipelineEntity needs to have many properties, many which are unused in certain components and required in others. Some properties will also have become redundant at the end of the pipeline.

For example, these components could be executed:

1. TransformCoordinatesComponent: Parse the raw coordinate data into a Coordinate type.
2. DetermineCountryComponent: Determine and stores country code.
3. DetermineOnRoadComponent: Determine and store whether coordinate is on a road.

Code:

pipeline
    .RegisterComponent(new TransformCoordinatesComponent())
    .RegisterComponent(new DetermineCountryComponent())
    .RegisterComponent(new DetermineOnRoadComponent());

pipeline.Process(positionPipelineEntity);

The PositionPipelineEntity type:

public class PositionPipelineEntity
{
    // Only relevant to the TransformCoordinatesComponent 
    public decimal RawCoordinateLatitude { get; set; }

    // Only relevant to the TransformCoordinatesComponent 
    public decimal RawCoordinateLongitude { get; set; }

    // Required by all components after TransformCoordinatesComponent 
    public Coordinate CoordinateLatitude { get; set; }

    // Required by all components after TransformCoordinatesComponent 
    public Coordinate CoordinateLongitude { get; set; }

    // Set in DetermineCountryComponent, not required anywhere.  
    // Requires CoordinateLatitude and CoordinateLongitude (TransformCoordinatesComponent)
    public string CountryCode { get; set; }

    // Set in DetermineOnRoadComponent, not required anywhere.  
    // Requires CoordinateLatitude and CoordinateLongitude (TransformCoordinatesComponent)
    public bool OnRoad { get; set; }
}

Problems:

• I'm very concerned about the dependency that a component has on properties. The way to solve this would be to create specific types for each component. The problem then is that I cannot chain them together like this.

• The other problem is the order of components in the pipeline matters. There is some dependency. The current structure does not provide any static or runtime checking for such a thing.

Any feedback would be appreciated.

Answer 1

It appears to me as though your problem here might be partly that your pipeline is a little too abstract. It's important to work with abstractions, but the openness of the design you have is making it hard for you to visualise the solution.

For example, with the order of components, you probably want to split these out in the pipeline, so that

pipeline
.RegisterComponent(new TransformCoordinatesComponent())
.RegisterComponent(new DetermineCountryComponent())
.RegisterComponent(new DetermineOnRoadComponent());

Might become something like

pipeline.TransformationCoordinates.Register(new TransformCoordinatesComponent());
pipeline.Determinations.Register(new DetermineCountryComponent());
pipeline.Determinations.Register(new DetermineOnRoadComponent());

Then your granularity becomes that of the number of different types of ordered component you need to have, so the Pipeline can have zero or many TransformationCoordinate components, Determination components and so on and will run all the ones available in order. Of course, this might be hidden inside pipeline.RegisterComponent splitting them out by interface, but this is an easier way to illustrate what I mean.

As you become less abstract you lose flexibility, in this case you would need to have a new collection if you added a new ordered type, so if you wanted to have an InvertComponent that ran between the TransformationCoordinate and the Determination components you would have to change your Pipeline accordingly. This is probably not a terrible thing, as long as your architecture is generally solid, however - you are likely to have to make the jump from the theoretical to the pragmatic at some point anyway.

Once you know the different component types you can guarantee that you have the correct properties available for each stage in the process- you could either ensure that the properties are set at each step on a type that always has them available or you could return a new object from each stage with the properties needed for the next.

Either way, I think you will find your work is easier if you start to think in more concrete terms about the exact processes you need to follow.