In machine learning we have modules that perform operations on data in a sequencial manner. Modules are generally the following:

  1. Data Collection Module: Takes raw data from a resource (filesystem,database,...) and ensures the data in the pipeline. We can assume that the result of this module is an object of type "Dataset"

  2. Data Cleaning Module: Takes as input the Dataset object and checks errors in data; mainly missing values and outputs a new Dataset object with clean data

  3. Data preprocessing: Takes the new Dataset object makes math operations on the data it wraps (normalization, normalization, standardization ...) and output a new dataset object with a new format.

  4. A Training Module: This module is responsible of training machine learning models using different algorithms, it can be on algorithm or multiple algorithms in multiple stage to have different result to compare them and select the best peroforming model

  5. Last, a test model: That takes the trained and selected model and ensures it has low error rates by inputting a sample of test data

The pipeline, we want to develop as you see encompasses a set of object, each of which performs a set of operations on our input data and passes the output to the next object in the sequence.So our data is forced to force the path we specify by configuring our pipeline. With some variable objects that can change in any of the chains of our pipeline.

The pipeline can be represented as follows:

raw_data ==> data_collection > data_cleaning > data_preprocessing > model_training > model_testing ==> model

These modules have some common operations such that execute() and validate() data. Imagine a single abstract class (let's say we call it IMLOperation) that works as an interface holding these common operations from which all the operations in the pipeline are derived (sub-classed as Preprocess, Collect .. objects). Do you think that this approach together with the Iterator Design Pattern to ensure the order is suitable to develop this pipeline? Or is the strategy pattern along with a client that provides a stack of ordered operations a better solution for this pipeline architecture ?

  • 1
    Not sure what you mean there. Wrap something in a lambda. Encapsulated. Also no need for interfaces when everything is duck typed. Take a read – Kain0_0 Feb 10 at 5:10
  • I modified the question and tried it as much as possible. I hope this conforms to the required format. – pentanol Feb 10 at 7:39
  • @DocBrown I reopened and edited the question to be more about Python suitability. Please let me know if you still have some concerns about the question. – maple_shaft Feb 10 at 14:31
  • @maple_shaft I understand the error I made when writing this question now. Thank you for the editing. – pentanol Feb 10 at 14:35
  • @DocBrown The issue is not about using python for machine learning or not. The idea is about designing a class diagram to automatize the flow of data in order to reach some kind of output(Model). This is sth similar of what I am asking about and want to achieve: spark.apache.org/docs/latest/ml-pipeline.html – pentanol Feb 10 at 18:23

As in any data processing or sequential operations, look for functional programming patterns as a cleaner alternative for design patterns.

Design patterns wise:

  • strategy pattern helps when you should pick 1 algorithm from a list of algorithms to apply as a transformation. Here your modules can be considered an algo to apply
  • iterator pattern here would be a glorified foreach loop over the individual data record (iteration unit)
  • map(reduce) pattern? = your operations here are a set of maps (as in apply a method in a chain https://en.wikipedia.org/wiki/Map_(higher-order_function)). When I have multiple transformation methods I tend to apply maps as it makes the code more modular and I can better test each part. Most mdoern programming languages are optimised for using maps to transform data, and the clarity gain outweights the performance gain from putting everything in 1 for loop in the long run (or any time for that matter, specially in a team)

Python implementation wise:

  • there is core method map that allows to chain operations in a lazy fashion
  • use a streaming framework (rxpy is what we use in production now) if you want to benefit from a great abstraction and helps with the general project organisation. Rxpy provides a framework, but there are no connectors out of the box afaik. Do use a framework for production work, either homebrewed or open source.

Python map example:

# raw data
a = [1,2,3,4]
# chain operations. i.e. basic lambdas - replace with your own module.run() 
b = map(lambda x: x*2, a)
c = map(lambda x: x-3, b)

# get the result as it's lazy in computing it
  • So if we follow what you mentioned, we can use the Strategy pattern to build ensemble learners that train multiple models or to train multiple models (ml agorithms) and choose the one with the best performance. But does the iterator enforce a certain order of operations in the ml pipeline along with iterating ? – pentanol Feb 11 at 13:54
  • Strategy patterns says that you pick 1 algo from a list of algos depending on a criteria (type of operation, metadata, etc). There is not actual software design pattern per se for training multiple patterns and picking one. This is just a scoring approach to picking the best model – Clau St Feb 12 at 15:24
  • What about a design pattern that represents the order operations in the pipeline. Is composite pattern for example a good way to represent the pipeline ? – pentanol Feb 13 at 6:01
  • 1
    The order itself isn't a pattern. Guess as patterns go, wanting to do something in order is imperative programing. I wouldn't try to worry to much about patterns for data processing, as there aren't many at a higher level, besides stream vs batch. Composite pattern is mostly useful for trees of data nodes (individual and collections). – Clau St Feb 15 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.