5

Scenario

We are building a UI that allows users to query our data in bulk. The return format is CSV and there is a decent amount of processing that needs to happen, so this processing processing is done in parallel and then sets of rows are streamed back to the UI.

Current Implementation

An AJAX POST request is sent to the server, it waits for all of the data to come in and then uses createObjectURL to trigger a download. The issue is that it can take several minutes for the request to complete and the user sees a spinning indicator indicating that their request is in progress.

Problem

It's a bad experience that the user cannot see any other progress and has no idea whether their request is actually processing or how long it will take.

Goal

To have the browser download the file as it's being streamed back to the client.

Obstacle

It does not seem possible to enable this behavior in JavaScript. The closest that I found was a script StreamSaver.js but using an external MITM is not acceptable and I don't understand the code well enough to determine whether this is a viable (or even good approach).

Other Ideas

  • Use a GET request or a form. The issue here is two-fold: (1) The request data can be nested and of variable length. It doesn't seem right to have to be parsing a request string on the backend. (2) It seems like this causes a page reload, which kills the state of the React application.

  • Instead of downloading the file continuously on the client, we can add a progress bar instead. To be this, we could add a header to the response indicating how many chunks are expected and then increment when each chunk is received. The downside of this approach is that the user cannot cancel the request and they cannot recover partial results.

Help

It feels like I'm missing a much better solution to this issue. Please feel free to challenge any of the assumptions above. My main goal is to get a clean solution that provides a good user experience.

Notes

The backend is a C# web application, React is being used on the front-end, and it's possible to pipe through a PHP layer if somehow that would help.

13
  • Use Skip() and Take() in Linq to section off the data into smaller pieces, and send those pieces in individual requests. Alternatively, just provide a file download; websites do this all the time, for very large files.
    – Robert Harvey
    Commented Mar 30, 2019 at 1:23
  • @RobertHarvey The data is being sent back to the client in chunks -- it's the client download portion that's proving troublesome. Is there a way to provide a file download that can be done in unison with generating the file? We'd prefer not to have to maintain a directory with old downloads, but this can be done if all of the other needs are met.
    – Jared Goguen
    Commented Mar 30, 2019 at 1:31
  • I really think that's probably the easiest and best solution. stackoverflow.com/q/4668906
    – Robert Harvey
    Commented Mar 30, 2019 at 1:32
  • I'm not sure if that solves the client issue, I'm using PushStreamContent to stream the file back to the client, but there is no way to download a stream from an AJAX request in JavaScript.
    – Jared Goguen
    Commented Mar 30, 2019 at 1:38
  • Well, if the problem is that the user gets a spinning indicator, why don't you load the data on a background thread and let the user do something else while it downloads?
    – Robert Harvey
    Commented Mar 30, 2019 at 1:57

1 Answer 1

Reset to default
2

I think what you are looking for is ReadableStream

https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/getReader

you can generate one from a fetch request and stream it as required

https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams

2
  • I'm getting a ReadableStream from the AJAX response but there does not appear to be a way to download it while it's still running. (Also, trying to use it to fuel a custom progress bar with download-at-the-end, but I'm having trouble transforming one chunk back into a string.)
    – Jared Goguen
    Commented Mar 30, 2019 at 16:57
  • the example seems to be doing it, you have to output the reads to another stream that you can listen to
    – Ewan
    Commented Mar 30, 2019 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.