-1

I'm developing an ETL process in Python and Pandas to pull data from a rest API, and then dump it into a relational database. A few of the fields that come back contain sensitive that I do not want to expose in the wider general purpose table.

I can run a drop column command and then not worry about the data. But I want to make the drop function more explicit that this is the removal of denied columns.

I'm thinking of making a wrapper or reference function for the drop column function in my own library that implements the exact same behavior, but with a name that calls out the explicit purpose.

Is this a reasonable design for making intentions about this operation clear? Is there another way I should make it clear that as part of delivering the final cleaned up data, all columns that should not be included have been intentionally removed from the final data set?

1 Answer 1

1

Yes, and Yes.

Yes you should absolutely take advantage of an abstracted interface to declare intentions, regardless of implementation. This makes it easier later down the track to swap in a different implementation when you for example realise that dropping columns doesn't work, and instead you need to generate pseudo random replacements.

Yes you could also make it clear via other approaches:

  • Comments,
  • Function names in the Business Logic controller (as opposed to the API),
  • Documentation.
2
  • Can you expand on Function names in the Business Logic controller? I am using an existing library, Pandas, to work with the information. It has a Drop function. I want to create a method in my own library that has the same signature that invokes the drop function. Is that not the same thing?
    – ADataGMan
    Commented Jun 17, 2020 at 14:01
  • 1
    Ah, I was under the belief you were discussing have a ScrubColumn method on the API and having the client call it, in which case having a named function wrap it would help document intention. Rereading your question my suggestion is in fact what you were describing, just on a different layer.
    – Kain0_0
    Commented Jun 17, 2020 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.