How to efficiently implement data-heavy screen updates in a web application?

Question

Imagine an engineering type web application where there is a screen that displays around 100 of unique data values. The user can "page through" the screen by clicking Right/Left arrows, and the screen is updated with the next page of 100 values. When the right page is found based on some specification, user can save that page to persistent storage.

How can I implement this technical-wise so that user interaction is fast?

Example of the app

What you see above is a Page. Page has 6 data blocks on it (not all are shown). Each data block contains 13 to 20 data points (those are the ~100 data points). User clicks to see next or previous page, and all the data points are updated with the new ones. When the right Page is found, you can save it (not shown).

Each data block on screen represents an entity in the database, so if need be I only need to pass one data id integer per block internally, where the data in the block can be retrieved from database.

Current Implementation

TL;DR: I pre-generate a huge data block of 6000 points, send it all to the user's browser, and feed it to JavaScript. Use pages through slices of data in that data block. This part is done in memory and going to the next page does not require roundtrip to the server.

Details:

1. pre-generate all blocks with all data points (there are around 60 pages to generate). 60 pages x 100 data points = 6000 data points to store
2. store that chunk data in the database during page generation process (for easier retrieval later via id)
3. use data from step 2 to generate the user screen via JavaScript. That is all 6000 points are held in the giant structure that is in browser memory (and is also on disk from step 2). Currently paging is implemented via JavaScript. A portion of the data from the giant data chunk is displayed when user pages through. It is supposed to be "fast" because it's all in memory, but it's decent, not crazy fast. Probably because the data is so huge.
4. User picks selection #X, and saves it. Upon saving, the database id and selection number are passed to the PHP script. using those, the correct page is loaded from previously saved record in step #2 and processed further.

Issues I am having

• data chunk is too big
• I store insanely huge records in the database and in memory
• it is not terribly fast despite currently being all in memory

Refactoring Idea

I am thinking that I can do AJAX-like data and instead of using JavaScript to page from one record to the next, use AJAX. The only downside is it will be heavily reliant on Internet connection, opposed to being all in memory like it is now. That could still slow it down. But the idea is that I could use AJAX to do a round-trip to the server for each new page and only load 100 points with each new request.. and not the giant 6000-point data record. But somewhere the application still needs to be aware that there is this huge data block.. I'm not entirely sure how I can indicate that there are going to be 60 pages (something I'll need to think about as currently they are all pregenerated into one huge chunk).

I am curious to find out if there are other (better) ways to do it, before I do the AJAX round-trip.

Question

Is there a more efficient way technically to implement this web application?

More info found

Turns out that JS Graphics library was slowing down page rendering - it renders a graph on each page, pulling graph points from the data. Removing graphics rendering divs sped up the paging process tremendously. In comparison, had I implemented paging using AJAX (without graphs), the "paging with Internet-roundtrip due to AJAX" would be slower in comparison to "current JS-paging without grapics".

It remains that I currently do things such as

1. print a 300+K JSON into a hidden DIV on html page
2. Use JS to page through the slices of the JSON

Before asking the question it seemed like passing this data into HTML page was overkill and wasteful. I deem that doing this is not so bad and that my performance-hit were the graph rendering and not the data.

I still think I can do better when saving data internally in one large chunk (including all pages). Perhaps I can instead save each page separately so that when user saves data, I can pull up that record only instead of the record with all the pages. But I'm not sure the effort to do so will necessarily be more efficient. So at the moment of writing I intend to leave things as is + look into page trimming.

Answer 1

You can use AJAX to load pages that are 3 pages-away from the current one. Eg. if you're on page 1 - issue request to fetch data for pages 1-3, if you click "next" you show data for page 2 (which are in memory), then you issue request to return pages 1-4...

On page 20 you'd request data for pages 17 to 23, etc.

Depends on if you want to have fresh data... but if that's not a concern you can just save each page's data in array and during AJAX calls - skip data you already have in browser's memory. So, to simplify you can just load data for first 3 pages, then when user click next - load data for page 4, etc. and users won't notice it's not realtime, as you'll be prefetching data.