I'm creating a web app using web.py (although I may later rewrite it for Tornado) which involves a lot of file manipulation. One example, the app will have a git-style 'commit' operation, in which some files are sent to the server and placed in a new folder along with the unchanged files from the last version. This will involve copying the old folder to the new folder, replacing/adding/deleting the files in the commit to the new folder, then deleting all unchanged files in the old folder (as they are now in the new folder).

I've decided on Heroku for the app hosting environment, and I am currently looking at cloud storage options that are built with these kinds of operations in mind. I was thinking of Amazon S3, however I'm not sure if that lets you carry out these kinds of file operations in-place. I was thinking I may have to load these files into the server's RAM and then re-insert them into the bucket, costing me a fortune. I was also thinking of Progstr Filer (http://filer.progstr.com/index.html) but that seems to only integrate with Rails apps.

Can anyone help with this? Basically I want file operations to be as cheap as possible.


I don't see any diff'ing or comparing between the files, so there's no need to copy them around or even load them into RAM.

My suggestion is: deal with the files individually. And if you want it to be really flexible, consider storing then in a database. If that's not feasible, at least only write the changed file to your storage. Store where's the newest version (and former versions) of that single file (not the folder) in your database.

Imagine one file is changed in a folder that contains 2k files. Just write the new file somewhere (or even better, most files are text and you can just write the delta), don't touch the entire folder.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.