4

I am looking to build an web-application which would allow user to upload csv file - see preview of few rows in the file and then apply different cleansing and standardisation rules.

For example the user can choose a telephone column and click a standardise button to format the phone numbers in a particular way. He can select state column and click enrich - which will add state based on zip code available in another column. All these transformations has be displayed in the preview table in real-time. He should also be able to do operations like undo, redo also.

My initial thought is to load the csv in back-end as a pandas dataframe and expose various functionality required in front end as api endpoints. But the problem here is user can undo operation he has done - how do I bring back the data frame to previous state. There could be many operations with multi level steps which could make this more complicated.

Is there a better solution for this ?

  • 1
    Take a look at OpenRefine it may already do the data manipulations etc. that you have in mind. You can also use python formulas to manipulate cells there. – Matthias Berth Mar 27 at 9:41
4

When an inverse command doesn't exist for every command there are two fundamental ways of implementing undo:

  • Reload the previous state

  • Replay all commands but the last over the initial state

For the second option to work the commands all need to be side effect free. That is, they should only be changing the state, nothing else. You can follow the command pattern here.

You can mix the options to tune your performance. Sometimes state is big and expensive. Sometimes commands are slow. It can be costly to store every state. It can take to much time to replay every command.

You can try to find a balance where you store the state every so many commands. This is called a space time trade off. Experimenting is the best way to find the right numbers for this balance.

Come to think of it, this is a bit like how Rainbow Tables work.

0

Yes. I'm assuming you want to use python libraries to process the data, you can use a javascript interpreter such as Brython to run python client side in the browser.

http://www.brython.info

However. I would recon the main part of such a project would be overcomming techincal difficulties getting all the technology working together, rather than coding the features of your app.

A faster solution would be to distribute a non web python application, or upload the file for processing on the server as you outline.

regarding the undo problem, I would simply store copies of the data at the various steps if you are only working on a small sample subset.

Otherwise, there is the command pattern. Store each step as a Command object, remember the order and which step the user is on, and you can rerun the steps on the data to get to the required step

  • The Python data science toolkit (numpy, scipy, pandas) cannot run on non-CPython implementations as they depend on native code. Server-side processing is the only option. – amon Mar 9 at 11:57
  • 1
    I didnt say it would be easy : droettboom.com/blog/2018/04/04/python-in-the-browser – Ewan Mar 9 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.