Optimization of a hybrid pagination scheme

Question

I'm working on a Web Application using node.js in which I'm building a partial copy of the database on the client-side to decrease the load on my server. Right now, I have a function like this (expressed as python-style pseudocode, but implemented in JavaScript):

get(table_name,primary_key):
    if primary_key in cache[table_name]:
        return cache[table_name][primary_key]
    else:
        x = get_data_from_server(table_name,primary_key) # socket.io
        return cache[table_name][primary_key] = x

While this scheme works perfectly well for caching individual rows, I'd like to extend it to support the creation of paginated tables ordered according to the primary_key, and loading additional data using the above function for only the current and possibly the adjacent pages. Now, I don't want to keep the list of primary keys on the server to be retrieved every time I need to change the page (which, for reasons beyond the scope here, will be very frequent), and keeping it on the client side, subject to real-time create/delete events from the server, doesn't seem that good an idea, even after compression (using ranges, instead of individual values).

What is the best way to calculate which items are to be displayed on a random page, minimizing the space requirements & the need for communication with the server?

Answer 1

Keep in mind that node is non blocking when waiting for io, not if you are processing items within your code. If you are using node to loop through or manipulate large datasets, consider offloading that work to the database or other processes.

It also sounds like you are trying to combine database logic (primary keys, rows) with application logic (user views, models) into a single caching function. Consider a dual approach of optimizing at the database layer, and the application layer:

At the database level, Look into your database usage, use EXPLAIN, examine your use of indexes, remove N+1 and other repeated database calls.

At the application level, think of your pages (especially on the initial load) as entire views instead of tables and rows. then cache all of the data needed for the most common views, if the same data is presented to every user, consider caching the entire rendered output. If using ajax calls, consider storing the initial set of cached data as json embedded within the html response. Also look at where your cache is stored, redis and memcached are popular datastores for live site data.

Ref: - NodeJS Event Loop: http://blog.mixu.net/2011/02/01/understanding-the-node-js-event-loop/ - MVVM architecture: http://addyosmani.com/blog/understanding-mvvm-a-guide-for-javascript-developers/ - Nodejs Page Caching: https://stackoverflow.com/questions/9715636/node-js-page-caching

Answer 2

If the server does the pagination isn't it already returning a small subset?

You could let the server return only an a list of primary-keys for the requested page. Materialize as many as you can from your cache, then bulk fetch the remaining identities from the server in one call.