Should I unit test "trivial" functions that are thing wrappers around the Pandas API?

Question

I'm working on a codebase with very little testing. The code is 90% an ETL pipeline with functions like

def sort_data(df: pd.DataFrame, column: str = 'date') -> pd.DataFrame:
   return df.sort_values(by=column).reset_index(drop=True)

or

def validate_no_duplicate_index(df: pd.DataFrame) -> pd.DataFrame:
   assert df.index.is_unique, 'Index is not unique'
   return df

Now, it's not particularly difficult to write tests for these functions, but it also seems to be a waste of time.

The real issue is that the non-trivial logic in the codebase is extremely reliant on these helper functions "doing their job".

def calculate_change(df: pd.DataFrame, date_column : str = 'date', value_column: str = 'value', lag=1) -> pd.Series:
    return df.pipe(sort_data, column=date_column )[value_column].diff(lag)

The astute reader might notice the code above has bugs - namely if df['date'] has duplicate or NaN values.

Any ideas on how to leverage unit testing in this scenario?

Answer 1

Does this test have value? That is: is there some behaviour that could reasonably be buggy, and is a bug so risky that writing a test is worth it? Risk is likelihood × impact. Tests are supposed to make the software development process take less effort overall.

Or in a simple formula that tells us whether writing a test that would detect a particular bug is worth it:

effort of writing the test < effort for dealing with the bug × likelihood of the bug

Consequently, quite a lot of tests are simply not worth writing. In particular:

• there might not be meaningful behaviour to test, as in your validate_no_duplicate_index() example
• bugs are very unlikely to occur
• bugs would have low impact, and in particular would be easy to debug and fix later, and would have little business impact
• testing that kind of behaviour would take disproportionate effort

Simple wrapper functions contain little behaviour and have a comparatively low chance of doing something wrong. Your functions do contain a bit of meaningful behaviour so they're not pure wrappers, but only you can judge whether there's enough risk in there to make the QA effort worth it. Perhaps these functions are so foundational in the code that if they were to have a bug, it would render anything else an undebuggable mess (high cost if there was a bug). I might then write tests just to have peace of mind, in order to write other parts of the software with more confidence.

It can also be noted that unit tests are only one kind of QA measure. There are other kinds of testing, like end-to-end tests that might be easier to set up in an ETL context. Just looking at the code can be a QA technique. Static analysis can find some bugs – for example, you're already using type annotations so you could use mypy to at least ensure that all methods you call exist, even if it doesn't understand subtleties of the Pandas API.

Sometimes you don't want so much upfront QA as improving your ability to debug problems later when they occur. For example, logging the progress of your code, dumping intermediate results, extracting expressions into named variables that can be inspected in a debugger, or checking your assumptions at run time with assertions. In a sense, such approaches reduce the right side of the above equation – the bugs become less costly if they occur.