I need to implement some system (using python pandas dataframe in my case) that parses raw data, then adds on calculated data, and then validates that calculated data (boolean output on some columns)
I couldn't make this problem simpler than what is below, so please bear with me.
Assumption: this is and always will be single threaded
The flow I think of implementing:
- Read raw data into a db
- Query the db, to obtain result (some other table)
- Parse the result (map or reduce operation, or multiple operations)
- Put parsed result back into original table, or into a new table.
- Validate result
I was thinking how to design this, and came up with the following:
- create a class that would hold all types of tables. It would allow for read and update operations only, on needed columns.
- create a static class for calculations on tables. Each query would take as arguments the tables it requires to make its calculations and return a new table.
- Create validator interface which has a method that takes in a table and outputs whether it is valid or not.
Now, I would do something like
db = DB()
df_raw = raw_parser.parse_raw()
calculation1 = Calculations.calculation1(df_raw)
calculation2 = Calculations.calculation2(df_raw, calculation1)
calculation3 = Calculations.calculation3(calculation1 , calculation2)
validator3 = Calculation3Validator(db.get_calculation3())
Is this a problem?
This doesn't seem very object oriented to me, because all the calculations sit together, statically, in Calculations.
Maybe it would have been smarter to somehow create a class for each calculated column, and assign it responsibility for calculating its own properties? Seems like too much of an abstraction, but I am not sure.
Is it ok to have a static class holding queries that accept and output tables, and that holds no state? Does it hold a risk of becoming a god class that can't be separated?
Is there a standard way of achieving my use case which I am very far from?
Sorry for the length of this question, I wanted to be very clear in my intent.