I am trying to think of the most sensible way to design the architecture of a data generation system with several steps.
Data in the system goes through several transformations which can be divided into separate steps (from the business logic point of view).
I would like the system to keep this modular design, in such a way that each module represents a step in the data transformation.
A module's input should be the previous module's output.
- What are some good ways to orchestrate this flow?
- How should modules communicate with each other?
- In each step, where should the input come from, and where should the output go?
- Is it a good idea to use a database as the source and target of data consumption / generation for each module?
- Should modules be built as separate scripts / executables which only directly communicate with the database?
The system will be implemented by several people. Each developer will be assigned a module. I would like the architecture to simplify the workflow by allowing each developer to work independently; and make assumptions only about the data their specific module consumes.
The modules relationship is depicted below.
Modules are represented as blue boxes. Some modules depend on data generated by other modules (black arrows). Some modules need to persist data on the DB (dotted gray arrows).