Data generation system modular design

Question

I am trying to think of the most sensible way to design the architecture of a data generation system with several steps.

Data in the system goes through several transformations which can be divided into separate steps (from the business logic point of view).

I would like the system to keep this modular design, in such a way that each module represents a step in the data transformation.

A module's input should be the previous module's output.

1. What are some good ways to orchestrate this flow?
2. How should modules communicate with each other?
3. In each step, where should the input come from, and where should the output go?
4. Is it a good idea to use a database as the source and target of data consumption / generation for each module?
5. Should modules be built as separate scripts / executables which only directly communicate with the database?

Edit:

The system will be implemented by several people. Each developer will be assigned a module. I would like the architecture to simplify the workflow by allowing each developer to work independently; and make assumptions only about the data their specific module consumes.

Edit 2:

The modules relationship is depicted below.

Modules are represented as blue boxes. Some modules depend on data generated by other modules (black arrows). Some modules need to persist data on the DB (dotted gray arrows).

Answer 1

Pipelines are extremely common paradigm in software design. They are quite common in scripting environments (most of the shells), where each step is separate program/process and functional world, where each step is a function. You should draw inspiration from them.

What are some good ways to orchestrate this flow? How should modules communicate with each other? In each step, where should the input come from, and where should the output go?

There is usually some third party that does all of this. It ensures each part is called in right order with right data.

Is it a good idea to use a database as the source and target of data consumption / generation for each module?

I believe modules should be independent on how the intermediate data is stored. It just gets data in some common format.

Should modules be built as separate scripts / executables which only directly communicate with the database?

They should all be separate. But they should receive the data, not pull it.

One thing that comes to mind for all of this is that you have two solutions for how to transfer data :

• Apply rigorous design of data structures, that are passed between modules, so all of them depend on those.
• Have each module create it's own data structure and have "transformation steps" between modules, that depend on data structures of modules between which this transformation is. This is common in shells, where you use text manipulation tools to transform output of one program into input of other.