Options to handle large (multi-gigabyte) file uploads

Question

How would you implement a very large file upload functionality with a Django application and S3?

In my side job as a photographer, I have several clients for which I have a need to share multi-gigabyte archive (zip, tar) files with that contain the originals as well as the processed images of the event in question. Up until now, we've been using Google Drive for this, in which I'd upload the file to GD and they'd retrieve and save in a local storage drive. I'd clean up the GD folder on occasion, since the files uploaded there are mirrored on my computer via the Mac Google Drive client. Given that my Mac only has a 256GB onboard drive, space is at a premium.

One client has had two hard drive failures over the past four months, where there were zero in the previous ten years I'd been employed by them. So they want a better solution, and I'm already a developer, so why not?

Now, the question is whether it is a good idea to have the browser be responsible for the queueing and transport of a twelve-plus gigabyte archive file to my server, for which it will go through some attribute scanning before being moved along to S3.

I can see two options with this:

1. Use the browser with a file upload form to upload a Multi-Part file to the server. Upon completion, the file will be checked and processed by a local Celery task, and then uploaded to a S3 bucket.
2. Implement a Django Management command to trigger execution of the local file processing Celery task, and use a local Python script using Paramiko to both upload the file and trigger execution of the management command when upload is complete.

Personally, I'm leaning towards Option 2, but would like other ideas if possible.

Answer 1

Through conversations with others about this topic, I think I've put together a solution:

1. Upload the archive file to S3
2. Upload action returns a S3 ID, which can be sent to an API endpoint
3. Server retrieves the file and passes to Celery task for processing.
4. OPTIONAL: email is sent to user/group for which the

In order to accomplish this, the following will have to be done:

1. Write a Python script to use Boto3's Multipart Upload
2. Boto3 method call will return a reference to the object, which can then be POSTed to a REST API endpoint
3. Server retrieves the file almost immediately over a fast fiber connection and starts an async job to process the time.