How would you implement a very large file upload functionality with a Django application and S3?
In my side job as a photographer, I have several clients for which I have a need to share multi-gigabyte archive (zip, tar) files with that contain the originals as well as the processed images of the event in question. Up until now, we've been using Google Drive for this, in which I'd upload the file to GD and they'd retrieve and save in a local storage drive. I'd clean up the GD folder on occasion, since the files uploaded there are mirrored on my computer via the Mac Google Drive client. Given that my Mac only has a 256GB onboard drive, space is at a premium.
One client has had two hard drive failures over the past four months, where there were zero in the previous ten years I'd been employed by them. So they want a better solution, and I'm already a developer, so why not?
Now, the question is whether it is a good idea to have the browser be responsible for the queueing and transport of a twelve-plus gigabyte archive file to my server, for which it will go through some attribute scanning before being moved along to S3.
I can see two options with this:
- Use the browser with a file upload form to upload a Multi-Part file to the server. Upon completion, the file will be checked and processed by a local Celery task, and then uploaded to a S3 bucket.
- Implement a Django Management command to trigger execution of the local file processing Celery task, and use a local Python script using Paramiko to both upload the file and trigger execution of the management command when upload is complete.
Personally, I'm leaning towards Option 2, but would like other ideas if possible.