1

I am trying to think of the most sensible way to design the architecture of a data generation system with several steps.

Data in the system goes through several transformations which can be divided into separate steps (from the business logic point of view).

I would like the system to keep this modular design, in such a way that each module represents a step in the data transformation.

A module's input should be the previous module's output.

  1. What are some good ways to orchestrate this flow?
  2. How should modules communicate with each other?
  3. In each step, where should the input come from, and where should the output go?
  4. Is it a good idea to use a database as the source and target of data consumption / generation for each module?
  5. Should modules be built as separate scripts / executables which only directly communicate with the database?

Edit:

The system will be implemented by several people. Each developer will be assigned a module. I would like the architecture to simplify the workflow by allowing each developer to work independently; and make assumptions only about the data their specific module consumes.

Edit 2:

The modules relationship is depicted below.

Modules are represented as blue boxes. Some modules depend on data generated by other modules (black arrows). Some modules need to persist data on the DB (dotted gray arrows).

Modules flow

  • Could you let us know what language you are developing in. The tools available in different languages can change the choices you make in designing something like this. – Matt Aug 11 '14 at 8:40
  • @Matt The technology is basically open for discussion. Although, since the question was asked, we settled on Python. Currently, we intend to implement each module as a Python module which exposes a 'run' method. All modules import a shared configuration module, which is initialized by a controller script. Other than initializing the config module, the controller also runs, according to the mentioned order, all other modules. Things like managing DB connection, loggers, etc. are also done by the controller. Any feedback and further advice will be much appreciated. – EyalAr Aug 11 '14 at 17:58
2

Pipelines are extremely common paradigm in software design. They are quite common in scripting environments (most of the shells), where each step is separate program/process and functional world, where each step is a function. You should draw inspiration from them.

What are some good ways to orchestrate this flow? How should modules communicate with each other? In each step, where should the input come from, and where should the output go?

There is usually some third party that does all of this. It ensures each part is called in right order with right data.

Is it a good idea to use a database as the source and target of data consumption / generation for each module?

I believe modules should be independent on how the intermediate data is stored. It just gets data in some common format.

Should modules be built as separate scripts / executables which only directly communicate with the database?

They should all be separate. But they should receive the data, not pull it.

One thing that comes to mind for all of this is that you have two solutions for how to transfer data :

  • Apply rigorous design of data structures, that are passed between modules, so all of them depend on those.
  • Have each module create it's own data structure and have "transformation steps" between modules, that depend on data structures of modules between which this transformation is. This is common in shells, where you use text manipulation tools to transform output of one program into input of other.
  • Thanks for the answer. I have a few concerns. Each module needs the complete data from the previous step. So they cannot be piped. I would need to wait for a module to complete its job before running the next step. Additionally, can you please give some ideas of how should modules get the data (if not from a DB)? What are some reasonable ways to transfer data (a lot of it) between modules? I can only think of a DB for this. Also, what are some ways of building a module to receive data instead of pulling it (assuming data can't be piped)? – EyalAr Aug 7 '14 at 10:54
  • @EyalAr Will each module need different "queries" over the data? Or are there common structures for all the data? Will structure of data change between modules? – Euphoric Aug 7 '14 at 12:02
  • Yes, modules will generally consume different data (some may be run parallely on the same input data). Yes, the structure of data will change between modules, as modules in one step, consume data generated by a module in the previous step. – EyalAr Aug 7 '14 at 12:23
  • please see my second edit to the question. – EyalAr Aug 7 '14 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.