I'm building a pipeline that scrapes data on entities, to keep things generic, let's call them Widgets
. Data on Widgets is currently unorganized and spread across various sources, including source code, spreadsheets, and issue/project tracking software like JIRA. The pipeline will run at some cadence and regularly update a table of data on Widgets.
I've designed the pipeline by breaking it up into small, individually-tested stages. These stages are linked together and the data they scrape/query is eventually joined together. For example, something like:
- Stage A scrapes data A.
- Stage B takes data A and retrieves data B.
- Stage C takes data A and retrieves data C.
- Stage D takes data B and retrieves data D.
- Stage E takes data A, B, C, D and joines it together into the finalized Widget data.
As you can tell, stages are dependent on one another. I've done this to avoid a stage needing to re-scrape/process data that has already been processed by a previous stage (e.g. stage B doesn't need to re-scrape data A, it instead operates on data A).
The main open design question currently keeping me up at night concerns stage E. I'm back-and-forth on how closely the finalized Widget data should mirror the existing structure of data A, B, C, D across the various sources. Additionally. how much processing/transforming should stage E do - if any - before creating the Widget data? As a simple example, if identification-related info exists within data A and data B, should my Widget schema have an identification
field that is composed of data from A and B, or should it have A
and B
fields that both hold identification
info. In other words:
Schema #1:
Widget {
A a {
string id
}
B b {
string name
}
}
Pros:
- Clear to consumers where data came from.
- Easier implementation, stage E basically just joins.
- Completely unopinionated w.r.t. how the
Widget
data will eventually be used.
Schema #2:
Widget {
Identification identification {
string id
string name
}
}
Pros:
- Easier to query for related info (e.g. all indentifcation info is in one field).
- Pipeline's structure doesn't "leak" into the Widget schema, changes to the pipeline's structure won't change the Widget schema (good for consumers of the Widget data).
I'm curious what the experts on SE think about this overall architecture and the question above. I'm guessing the answer will be "it depends", but I'm interested in at least more pros/cons to the approaches above. Links to any learning resources are also greatly appreciated!