Designing .NET C# service endpoint for downloading big files

Question

I have a UI and a .NET C# backend server that communicates with each other. I have a route /getBigFile that the UI calls on a button click, which the C# backend can take up to 60 seconds to create and return so the user can download it. (The file size can be anywhere between 1 kB to 600 MB.)

This would mean that the UI would have to wait with an open connection until C# returns the HttpResponse. This is obviously a bad design as it forces the user to stay on the webpage until the file comes back.

I thought of different solutions to go about this:

• Stream the file to the UI so the download begins right away.

• Have the C# backend return a httpResponse right away after launching an async job that creates the file, uploads it to S3 and emails the file upon success/failure. (Will the async job part even work if the client server connection is closed)

I am not sure which way would be better. The first way seems like the better UX choice, but when multiple users request big files, I feel like second choice will be handled more nicely if I implement it with a Queuing mechanism (using Amazon SQS or Redis. I don't really know how this will work though.)

I have never had to design something like this and any pointers and resources/guidance would be greatly appreciated!

Answer 1

If the creation of the file is progressive, then you don't need to wait 60 seconds before starting the download. Just send the file to the user as you are generating it, saving up to 60 seconds to your users.

This would apply to many processes such as encryption, compression, video conversion, etc. A problematic case would be the one where you have to process the whole file in order to know which would be its first bytes (for instance if the format of the file requires to specify either its length or its hash). If this is your case, see if there is a way to circumvent the limitation.

Note that in some cases, you'll even be able to specify the size of the file before sending it to the user (and if you can do it, it's a very nice feature, in order for the web browser to show the progress bar while downloading the file). For instance, many encryption algorithms return as many bytes as the input, making it straightforward to determine the actual size of the resulting file.

Otherwise, the HTTP POST which starts processing the file should return just an ID which makes it possible to track the progression of the first stage. Once the first stage is finished and the file can be downloaded, another route should make it possible to download the file using the original ID. This can be used easily with AJAX polling to let the user wait before downloading the file, and then present him with a download link.