4

I'm developing an application where it read data from different data sources. And then those data should be pre-processed and then go through some chain of steps (Filters ?) where those data will get processed and augmented. Finally those data should be written to a common data base.

I'm thinking of a Pipe and Filter kind of style to implement this. While I'm learning on this, I came across these invariants of this style [here].

Independent entities
---- Do not share state
---- Have no knowledge of other filters

Transformation
---- Incremental
---- Not dependent on order in the chain

And I'm having trouble understanding these. Why those are considered as invariants. ?

What happen if they share the state.
what happen if they have a knowledge of other filters.
What if they need to depend on other filters (In my case a pre-process is a must) and what is Incremental.

As I know violating the invariants might erode the code with the time. So if I use this Pipe and Filter style in my app, what kind of things will violate these invariants or what are the things that I can do to violate these invariants ?

Can anyone help me on this.

11
  • You might also want to read up on Monads. Dec 26, 2017 at 18:56
  • 1
    To be precise, for pipes and filters, the invariant should be "do not share mutable state". If there is some immutable state like a configuration passed into several filters, and they all share it, this does not break the architecture.
    – Doc Brown
    Dec 26, 2017 at 19:39
  • @DocBrown Can you please share some idea to share the state among the Filters. As I comment on the answer from Telastyn. (In my application people can add custom Filters (which are actually scripts which follows a basic pattern). So then I need to make sure they don't violate any invariants. Knowing how to violate might help me to catch the violation. Just to know, How can someone share the state between the Filters anyway (know it's not a good thing but just want to know how to do that))
    – prime
    Dec 26, 2017 at 19:42
  • @prime: I already gave an example, so what are you asking for?
    – Doc Brown
    Dec 26, 2017 at 20:25
  • 1
    @FilipMilovanović: that's the problem with this word - "dependency" means different things in different contexts, and specificially here, using this word for an explanation can actually confuse things ;-) And by the way, in several UML diagrams, a dependency arrow will exactly mean only "dependency from another component/piece of code/class", not "shared mutual state".
    – Doc Brown
    Dec 27, 2017 at 10:09

1 Answer 1

4

Why those are considered as invariants. ?

Because they're the intrinsic things that define this sort of architecture. If you allow shared state or knowledge of filters, you're no longer doing Pipe and Filter, you're doing something else.

The entire point of this architecture is to create independent, composable, parallelizable streams of work. This allows you to optimize the processing via reordering the filters. This allows you great scale, since the entity processing can be farmed out to many machines. It allows easy development since each filter can be implemented (and tested, and deployed) in isolation. And since these rules are uniform for all entities and all filters, it allows you to make high quality tooling for using them.

As soon as you start making exceptions to these rules, you start losing the benefits of having them. If you share state, then your entities cannot be trivially parallelized. If your filters depend on order, you can't optimize them, and you can cause subtle errors due to the implicit dependency.

5
  • Things getting clear now. Thanks for the detailed answer. In my application people can add custom Filters (which are actually scripts which follows a basic pattern). So then I need to make sure they don't violate any invariants. Knowing how to violate might help me to catch the violation. Just to know, How can someone share the state between the Filters anyway (know it's not a good thing but just want to know how to do that).
    – prime
    Dec 26, 2017 at 19:37
  • 2
    @prime - If they're literally scripts (arbitrary code) then there's literally an infinite number of ways they can share state between themselves. Without knowing more, I can't say for sure.
    – Telastyn
    Dec 26, 2017 at 19:40
  • Well one script can do a single thing. Lets say script is supposed to read a log file and filter out specific data set from that log. And there are many data types and filtering them are different. So we need multiple scripts. So what kind of state can be shared among the Filters and how can it be done. (asking how to violate the P&F) Couple of simple examples will help me a lot.
    – prime
    Dec 26, 2017 at 19:44
  • @prime - if multiple scripts share that log file, that's by definition shared state.
    – Telastyn
    Dec 26, 2017 at 20:25
  • what if they don't edit or change the log file but just read it only. Will that be violating the architecture ?
    – prime
    Dec 27, 2017 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.