Imagine an engineering type web application where there is a screen that displays around 100 of unique data values. The user can "page through" the screen by clicking Right/Left arrows, and the screen is updated with the next page of 100 values. When the right page is found based on some specification, user can save that page to persistent storage.

How can I implement this technical-wise so that user interaction is fast?

Example of the app

enter image description here

What you see above is a Page. Page has 6 data blocks on it (not all are shown). Each data block contains 13 to 20 data points (those are the ~100 data points). User clicks to see next or previous page, and all the data points are updated with the new ones. When the right Page is found, you can save it (not shown).

Each data block on screen represents an entity in the database, so if need be I only need to pass one data id integer per block internally, where the data in the block can be retrieved from database.

Current Implementation

TL;DR: I pre-generate a huge data block of 6000 points, send it all to the user's browser, and feed it to JavaScript. Use pages through slices of data in that data block. This part is done in memory and going to the next page does not require roundtrip to the server.

Details:

  1. pre-generate all blocks with all data points (there are around 60 pages to generate). 60 pages x 100 data points = 6000 data points to store
  2. store that chunk data in the database during page generation process (for easier retrieval later via id)
  3. use data from step 2 to generate the user screen via JavaScript. That is all 6000 points are held in the giant structure that is in browser memory (and is also on disk from step 2). Currently paging is implemented via JavaScript. A portion of the data from the giant data chunk is displayed when user pages through. It is supposed to be "fast" because it's all in memory, but it's decent, not crazy fast. Probably because the data is so huge.
  4. User picks selection #X, and saves it. Upon saving, the database id and selection number are passed to the PHP script. using those, the correct page is loaded from previously saved record in step #2 and processed further.

Issues I am having

  • data chunk is too big
  • I store insanely huge records in the database and in memory
  • it is not terribly fast despite currently being all in memory

Refactoring Idea

I am thinking that I can do AJAX-like data and instead of using JavaScript to page from one record to the next, use AJAX. The only downside is it will be heavily reliant on Internet connection, opposed to being all in memory like it is now. That could still slow it down. But the idea is that I could use AJAX to do a round-trip to the server for each new page and only load 100 points with each new request.. and not the giant 6000-point data record. But somewhere the application still needs to be aware that there is this huge data block.. I'm not entirely sure how I can indicate that there are going to be 60 pages (something I'll need to think about as currently they are all pregenerated into one huge chunk).

I am curious to find out if there are other (better) ways to do it, before I do the AJAX round-trip.

Question

Is there a more efficient way technically to implement this web application?

More info found

Turns out that JS Graphics library was slowing down page rendering - it renders a graph on each page, pulling graph points from the data. Removing graphics rendering divs sped up the paging process tremendously. In comparison, had I implemented paging using AJAX (without graphs), the "paging with Internet-roundtrip due to AJAX" would be slower in comparison to "current JS-paging without grapics".

It remains that I currently do things such as

  1. print a 300+K JSON into a hidden DIV on html page
  2. Use JS to page through the slices of the JSON

Before asking the question it seemed like passing this data into HTML page was overkill and wasteful. I deem that doing this is not so bad and that my performance-hit were the graph rendering and not the data.

I still think I can do better when saving data internally in one large chunk (including all pages). Perhaps I can instead save each page separately so that when user saves data, I can pull up that record only instead of the record with all the pages. But I'm not sure the effort to do so will necessarily be more efficient. So at the moment of writing I intend to leave things as is + look into page trimming.

  • 4
    Amongst other issues, your application seems to be missing a filter function (what you state as "when the right page is found"). You are currently leaving this filtering to the end user which is not very friendly. Analyse how the user finds the "right page" and offer server-side filtering based on that criteria. If done right, this will reduce the sizes of the chunks sent to the browser massively. – tofro Feb 13 '17 at 7:58
  • 1
    .. and if that is true, AJAX won't help you. I would indeed try to optimize the local display speed. Programs don't become slow just "because the data is so huge", they become slow if they have to process all that data. Did you use proper in-memory indexing by utilizing dictionaries in your Javascript rendering routine, to make sure you only have to process the data which is visible on the current page? – Doc Brown Feb 13 '17 at 9:46
  • 1
    What framework/librairies do you use in the client-side ? Some things like angularJS are not made to handle that much. Also if you have all in-memory and your rendering is not fast, the way it is implemented is probably not efficient. Also, can you check with the debogguer from chrome the timing ? It will show you how long take the server's call, how long take the rendering and so on. SO you will know what is really slow. – Walfrat Feb 13 '17 at 12:25
  • 1
    If 6000 data points need 300kB, you have ~50bytes per point - this seems to be a huge overhead if a data point is often just an integer or float. Anyway, I guess this cannot easily be solved on a conceptual level. One must probably look into your real code, profile and optimize that. Stackoverflow or Codereview.stackexchange might be suited for this. – Doc Brown Feb 13 '17 at 22:23
  • 1
    Ah indeed........... taking away JS graph library rendering divs does speed things up tremendously. I think that may be my issue. Today's lesson might be avoid using graphics libraries when you can. The culprit is/was the graphics rendering. As such, using AJAX will not naturally help (because without graphs, paging works incredibly fast, to where AJAX would struggle to keep up outside of LAN). The downside is.. still having a huge data structure, but I don't see a better way. – Dennis Feb 13 '17 at 22:24

You can use AJAX to load pages that are 3 pages-away from the current one. Eg. if you're on page 1 - issue request to fetch data for pages 1-3, if you click "next" you show data for page 2 (which are in memory), then you issue request to return pages 1-4...

On page 20 you'd request data for pages 17 to 23, etc.

Depends on if you want to have fresh data... but if that's not a concern you can just save each page's data in array and during AJAX calls - skip data you already have in browser's memory. So, to simplify you can just load data for first 3 pages, then when user click next - load data for page 4, etc. and users won't notice it's not realtime, as you'll be prefetching data.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.