In machine learning we have modules that perform operations on data in a sequencial manner. Modules are generally the following:
Data Collection Module: Takes raw data from a resource (filesystem,database,...) and ensures the data in the pipeline. We can assume that the result of this module is an object of type "Dataset"
Data Cleaning Module: Takes as input the Dataset object and checks errors in data; mainly missing values and outputs a new Dataset object with clean data
Data preprocessing: Takes the new Dataset object makes math operations on the data it wraps (normalization, normalization, standardization ...) and output a new dataset object with a new format.
A Training Module: This module is responsible of training machine learning models using different algorithms, it can be on algorithm or multiple algorithms in multiple stage to have different result to compare them and select the best peroforming model
Last, a test model: That takes the trained and selected model and ensures it has low error rates by inputting a sample of test data
The pipeline, we want to develop as you see encompasses a set of object, each of which performs a set of operations on our input data and passes the output to the next object in the sequence.So our data is forced to force the path we specify by configuring our pipeline. With some variable objects that can change in any of the chains of our pipeline.
The pipeline can be represented as follows:
raw_data ==> data_collection > data_cleaning > data_preprocessing > model_training > model_testing ==> model
These modules have some common operations such that
validate() data. Imagine a single abstract class (let's say we call it
IMLOperation) that works as an interface holding these common operations from which all the operations in the pipeline are derived (sub-classed as Preprocess, Collect .. objects). Do you think that this approach together with the Iterator Design Pattern to ensure the order is suitable to develop this pipeline? Or is the strategy pattern along with a client that provides a stack of ordered operations a better solution for this pipeline architecture ?