0

We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV.

Currently the above use case is handled in the following steps,

  1. User views a report and she clicks on download.
  2. The browser makes an API call, and on the server we save the output of the same SQL query in a csv file. (Using some trivial logics)
  3. We save the file on a distributed gluster FS. This fs is mounted on all the prod instances.
  4. And we return a download link after the above operations as the response of the API call.
  5. Client side javascript logic reads the download URL from the response and triggers a get request on it.

I think the above approach is too synchronous and gluster is becoming an overhead to maintain.

We generally have lots of download requests, but each file is around ~2-3KB.

Does the above approach is the most optimal way to solve this problem? I also wanted to get some ideas on how Google Analytics provides this same kind of feature. Is there a way I can remove gluster while still not overloading the client to generate a CSV?

Edit:

GlusterFS is a distributed file system which is mounted on our prod servers. We have configured it to replicate, so if any of the instance writes on the mounted path, the changes are available on all the instances on the same path.

We are facing issues with this approach of mounting and maintaining this distributed file system. We face issues like gluster gets unmounted on one of the many prod instances and if the load balancer directs requests over to that instance, it raises errors.

7
  • If you think your approach is too synchronous, why not make the API calls asynchronous?
    – Doc Brown
    May 25, 2021 at 15:32
  • How long do these reports take to create? May 25, 2021 at 17:45
  • @GregBurghardt The reports generally take ~500ms - 1s to generate reports. Based on the time period the user selects, the size can range from KBs to MBs.
    – phraniiac
    May 26, 2021 at 5:25
  • @DocBrown It makes sense to make the JS calls async and we have that in our refactoring plan. I was more thinking about the glusterfs approach and if we can do better in that part.
    – phraniiac
    May 26, 2021 at 5:26
  • 1
    Honestly if a report takes only one second to generate, writing the results to a file is like hitting a thumb tack with a sledge hammer. Refactor the ajax calls to be asychronous and pass the data back to the client in the response body. Keep it simple. May 26, 2021 at 13:14

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.