Pipeline's outputted data mirroring pipeline's architecture

Question

I'm building a pipeline that scrapes data on entities, to keep things generic, let's call them Widgets. Data on Widgets is currently unorganized and spread across various sources, including source code, spreadsheets, and issue/project tracking software like JIRA. The pipeline will run at some cadence and regularly update a table of data on Widgets.

I've designed the pipeline by breaking it up into small, individually-tested stages. These stages are linked together and the data they scrape/query is eventually joined together. For example, something like:

• Stage A scrapes data A.
• Stage B takes data A and retrieves data B.
• Stage C takes data A and retrieves data C.
• Stage D takes data B and retrieves data D.
• Stage E takes data A, B, C, D and joines it together into the finalized Widget data.

As you can tell, stages are dependent on one another. I've done this to avoid a stage needing to re-scrape/process data that has already been processed by a previous stage (e.g. stage B doesn't need to re-scrape data A, it instead operates on data A).

The main open design question currently keeping me up at night concerns stage E. I'm back-and-forth on how closely the finalized Widget data should mirror the existing structure of data A, B, C, D across the various sources. Additionally. how much processing/transforming should stage E do - if any - before creating the Widget data? As a simple example, if identification-related info exists within data A and data B, should my Widget schema have an identification field that is composed of data from A and B, or should it have A and B fields that both hold identification info. In other words:

Schema #1:

Widget {
  A a {
    string id
  }
  B b {
    string name
  }
}

Pros:

• Clear to consumers where data came from.
• Easier implementation, stage E basically just joins.
• Completely unopinionated w.r.t. how the Widget data will eventually be used.

Schema #2:

Widget {
  Identification identification {
    string id
    string name
  }
}

Pros:

• Easier to query for related info (e.g. all indentifcation info is in one field).
• Pipeline's structure doesn't "leak" into the Widget schema, changes to the pipeline's structure won't change the Widget schema (good for consumers of the Widget data).

I'm curious what the experts on SE think about this overall architecture and the question above. I'm guessing the answer will be "it depends", but I'm interested in at least more pros/cons to the approaches above. Links to any learning resources are also greatly appreciated!

Answer 1

I would propose a slightly different architecture, that would also avoid accidentally exposing the pipeline nature of the data processing.

There are 2 data structures

1. intermediate Widget data: This contains all the Widget data that is collected and/or generated during the processing. Some of the data might be only relevant for the communication between the stages of processing.
2. finalized Widget data: This is the data structure that emerges at the end of the pipeline and it only contains the data that is intended to be offered to the clients/users.

The pipeline steps would be amended like this

• Stage A scrapes data A and puts it in the intermediate Widget data.
• Stage B takes the intermediate Widget data and augments it with data B.
• Stage C takes the intermediate Widget data and augments it with data C.
• Stage D takes the intermediate Widget data and augments it with data D.
• Stage E takes the intermediate Widget data and filters it into the finalized Widget data.

As every stage works on the intermediate Widget data, it becomes less important which stage provides which data and it is easier to add a new additional stage later on. There is still a dependency between the stages, for example, because stage D needs data that was filled earlier, but the implementation of stage D does not have to know a stage-specific data structure to be able to retrieve what it needs.

At the point where you construct your pipeline, you need to know what each stage needs and provides so that you can construct it in the correct order.

The advantage is that you can split or rearrange stages in your pipeline without affecting the stages further downstream. If, for example, you find that some work done by stages B and C is actually duplicated (but with different fields being filled as a result), you can refactor that duplicated work to a new stage BC and stage D (which uses one of those fields filled by the duplicated work) would not be affected in any way.