0

I need to implement some system (using python pandas dataframe in my case) that parses raw data, then adds on calculated data, and then validates that calculated data (boolean output on some columns)

I couldn't make this problem simpler than what is below, so please bear with me.

Assumption: this is and always will be single threaded

The flow I think of implementing:

  1. Read raw data into a db
  2. Query the db, to obtain result (some other table)
  3. Parse the result (map or reduce operation, or multiple operations)
  4. Put parsed result back into original table, or into a new table.
  5. Validate result

I was thinking how to design this, and came up with the following:

  1. create a class that would hold all types of tables. It would allow for read and update operations only, on needed columns.
  2. create a static class for calculations on tables. Each query would take as arguments the tables it requires to make its calculations and return a new table.
  3. Create validator interface which has a method that takes in a table and outputs whether it is valid or not.

Now, I would do something like

db = DB()
df_raw = raw_parser.parse_raw()
db.add_raw(df_raw)
calculation1 = Calculations.calculation1(df_raw)
calculation2 = Calculations.calculation2(df_raw, calculation1)
calculation3 = Calculations.calculation3(calculation1 , calculation2)
db.add_calculation3(calculation3)
validator3 = Calculation3Validator(db.get_calculation3())
validator3.validate()

Is this a problem?

This doesn't seem very object oriented to me, because all the calculations sit together, statically, in Calculations.

Maybe it would have been smarter to somehow create a class for each calculated column, and assign it responsibility for calculating its own properties? Seems like too much of an abstraction, but I am not sure.


Question

Is it ok to have a static class holding queries that accept and output tables, and that holds no state? Does it hold a risk of becoming a god class that can't be separated?

Is there a standard way of achieving my use case which I am very far from?


Sorry for the length of this question, I wanted to be very clear in my intent.

  • Writing to the DB, Querying the DB, processing the data and then putting the result back into the database sounds rather long-winded. Can't you do all the map/reduce logic before inserting it into the DB in the first place? – Ben Cottrell Jun 25 at 12:23
  • @BenCottrell The point in all this, I didn't specify, is creating a db, ready for future queries at a moment's notice. I need to have an extremely easy way of implementing new queries on the db, which can depend on old queries. I am not performance or space bound, and I believe, correct me if I'm wrong, saving everything would be the fastest way for this requirement. The expensive resource here is HUMAN TIME of understanding what I already have and creating new insights of it – Gulzar Jun 25 at 12:35
5

This doesn't seem very object oriented to me

So what? Just because you can write OO code, doesn't mean you should. Use the approach best suited to the task.

Is it ok to have a static class holding queries that accept and output tables, and that holds no state?

Absolutely. This is exactly what static classes are for … in languages that are completely class/object based. But you say you are using python. That supports "modules" of functions I believe, so is a class even needed in your case?

Does it hold a risk of becoming a god class that can't be separated?

I can't see how in your case. They are static functions, so could be split out into other classes etc with a bit of refactoring. It's when you start using state that you will have problems here.

If you have a function that is stateless and has no side effects (eg doesn't directly read from a database or modify the parameters), then its a pure function. So mark it as such in a way appropriate to the language, eg by putting it in a static class.

  • 1
    @BenCottrell A real question that may sound dumb - Why isn't a DB+pure functions global variables in disguise? Where's the encapsulation/modularity in that? – Gulzar Jun 25 at 12:38
  • 2
    @Gulzar As per the T. Sar's comment, the main difference is really that the database is an external service. Global variables tend to cause problems with the structure and flow of the code because they represent an internal shared state which couples the code together, but also whose lifetime is the same as the lifetime of the running process, which generally makes it harder to observe and debug. – Ben Cottrell Jun 25 at 16:03
  • 1
    @BenCottrell Layered testing solves that. If your static function is well tested, you have no need to mock it out - you have another part of your testing block assuring it works. You can consider it part of the underlying API and call it a day, worrying only about testing the rest of the method that uses it. It turns the design of the code way more simple and kills a bunch of uneeded complexity - and the issues that come with it - that a purist "mock everything" methodology ends up adding. – T. Sar Jun 25 at 16:33
  • 1
    Just to add to @T.Sar's comment there, pure functions should never need mocking. They are deterministic in nature, so just let code under test call them. Remember, the "unit" in unit tests isn't some isolated unit of code, its isolated in terms of it has no dependencies that can have side effects. Mocking is only needed for parts of the code that must have side effects. – David Arno Jun 25 at 17:31
  • 1
    @DavidArno Or to substitute somewhat non-trivial inputs in parts that are specially hard to emulate, like logging systems or external webservices! – T. Sar Jun 25 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.