I'm building a pipeline that scrapes data on entities, to keep things generic, let's call them Widgets. Data on Widgets is currently unorganized and spread across various sources, including source code, spreadsheets, and issue/project tracking software like JIRA. The pipeline will run at some cadence and regularly update a table of data on Widgets.

I've designed the pipeline by breaking it up into small, individually-tested stages. These stages are linked together and the data they scrape/query is eventually joined together. For example, something like:

  • Stage A scrapes data A.
  • Stage B takes data A and retrieves data B.
  • Stage C takes data A and retrieves data C.
  • Stage D takes data B and retrieves data D.
  • Stage E takes data A, B, C, D and joines it together into the finalized Widget data.

As you can tell, stages are dependent on one another. I've done this to avoid a stage needing to re-scrape/process data that has already been processed by a previous stage (e.g. stage B doesn't need to re-scrape data A, it instead operates on data A).

The main open design question currently keeping me up at night concerns stage E. I'm back-and-forth on how closely the finalized Widget data should mirror the existing structure of data A, B, C, D across the various sources. Additionally. how much processing/transforming should stage E do - if any - before creating the Widget data? As a simple example, if identification-related info exists within data A and data B, should my Widget schema have an identification field that is composed of data from A and B, or should it have A and B fields that both hold identification info. In other words:

Schema #1:

Widget {
  A a {
    string id
  B b {
    string name


  • Clear to consumers where data came from.
  • Easier implementation, stage E basically just joins.
  • Completely unopinionated w.r.t. how the Widget data will eventually be used.

Schema #2:

Widget {
  Identification identification {
    string id
    string name


  • Easier to query for related info (e.g. all indentifcation info is in one field).
  • Pipeline's structure doesn't "leak" into the Widget schema, changes to the pipeline's structure won't change the Widget schema (good for consumers of the Widget data).

I'm curious what the experts on SE think about this overall architecture and the question above. I'm guessing the answer will be "it depends", but I'm interested in at least more pros/cons to the approaches above. Links to any learning resources are also greatly appreciated!

  • Is there a table that provides one-to-one mapping between "A.id" and "B.name" ? – rwong Mar 10 at 19:23
  • No, Widget would be the first. – sir_thursday Mar 10 at 19:25
  • 1
    Re-organise it into the shape of data for communicating to the user. If that means a flat struct, or a nested tree then so be it. – Kain0_0 Mar 10 at 22:46

I would propose a slightly different architecture, that would also avoid accidentally exposing the pipeline nature of the data processing.

There are 2 data structures

  1. intermediate Widget data: This contains all the Widget data that is collected and/or generated during the processing. Some of the data might be only relevant for the communication between the stages of processing.
  2. finalized Widget data: This is the data structure that emerges at the end of the pipeline and it only contains the data that is intended to be offered to the clients/users.

The pipeline steps would be amended like this

  • Stage A scrapes data A and puts it in the intermediate Widget data.
  • Stage B takes the intermediate Widget data and augments it with data B.
  • Stage C takes the intermediate Widget data and augments it with data C.
  • Stage D takes the intermediate Widget data and augments it with data D.
  • Stage E takes the intermediate Widget data and filters it into the finalized Widget data.

As every stage works on the intermediate Widget data, it becomes less important which stage provides which data and it is easier to add a new additional stage later on. There is still a dependency between the stages, for example, because stage D needs data that was filled earlier, but the implementation of stage D does not have to know a stage-specific data structure to be able to retrieve what it needs.

At the point where you construct your pipeline, you need to know what each stage needs and provides so that you can construct it in the correct order.

The advantage is that you can split or rearrange stages in your pipeline without affecting the stages further downstream. If, for example, you find that some work done by stages B and C is actually duplicated (but with different fields being filled as a result), you can refactor that duplicated work to a new stage BC and stage D (which uses one of those fields filled by the duplicated work) would not be affected in any way.

  • Thanks for the suggestion Bart. You stated "As every stage works on the intermediate Widget data, it becomes less important which stage needs which data" - I'm not convinced this is the case, as the intermediate Widget data changes as the pipeline progresses through stages, meaning the dependencies between stages are still present. For example, with your proposed architecture, stage D implicitly still depends upon stage B (and needs stage B to run first). It's not totally clear to me yet what benefits your proposal gives. – sir_thursday Mar 11 at 16:04
  • @sir_thursday, I have updated my answer – Bart van Ingen Schenau Mar 12 at 7:02
  • Thanks for the update. I'll think over your proposal a bit more. I'll give other experts some time to respond, and if there are no additional responses by next week, I'll accept yours. Thanks again! – sir_thursday Mar 12 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.