8

How would you implement a very large file upload functionality with a Django application and S3?

In my side job as a photographer, I have several clients for which I have a need to share multi-gigabyte archive (zip, tar) files with that contain the originals as well as the processed images of the event in question. Up until now, we've been using Google Drive for this, in which I'd upload the file to GD and they'd retrieve and save in a local storage drive. I'd clean up the GD folder on occasion, since the files uploaded there are mirrored on my computer via the Mac Google Drive client. Given that my Mac only has a 256GB onboard drive, space is at a premium.

One client has had two hard drive failures over the past four months, where there were zero in the previous ten years I'd been employed by them. So they want a better solution, and I'm already a developer, so why not?

Now, the question is whether it is a good idea to have the browser be responsible for the queueing and transport of a twelve-plus gigabyte archive file to my server, for which it will go through some attribute scanning before being moved along to S3.

I can see two options with this:

  1. Use the browser with a file upload form to upload a Multi-Part file to the server. Upon completion, the file will be checked and processed by a local Celery task, and then uploaded to a S3 bucket.
  2. Implement a Django Management command to trigger execution of the local file processing Celery task, and use a local Python script using Paramiko to both upload the file and trigger execution of the management command when upload is complete.

Personally, I'm leaning towards Option 2, but would like other ideas if possible.

  • 3
    I am not familliar with the details of Django and Celery, but whatever solution you choose, you should probably make it so robust you can continue a partially successful upload at later time after an interruption. Utilizing multi-part archives might be a good idea for this, together with checksums for each part. – Doc Brown Feb 19 '17 at 15:29
  • I would try to resist the temptation to re-invent rsync, though. Seems like it is the tool for your job. – 5gon12eder Feb 22 '17 at 13:52
1

Through conversations with others about this topic, I think I've put together a solution:

  1. Upload the archive file to S3
  2. Upload action returns a S3 ID, which can be sent to an API endpoint
  3. Server retrieves the file and passes to Celery task for processing.
  4. OPTIONAL: email is sent to user/group for which the

In order to accomplish this, the following will have to be done:

  1. Write a Python script to use Boto3's Multipart Upload
  2. Boto3 method call will return a reference to the object, which can then be POSTed to a REST API endpoint
  3. Server retrieves the file almost immediately over a fast fiber connection and starts an async job to process the time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.