How to do pagination while page gets populated dynamically

Question

I have a page where I list some products based on search data. The products shown are not saved in local database, but are taken live using API (and this is how it has to stay).

When page loads, it first lists products from one provider (first API call). After page finishes loading, then with Ajax the 2nd call to 2nd provider is made, product list comes and is appended to the page.

Considering each api call takes 2+ seconds to return data, I can't leave more than 1 call on page load.

My question is: How to do pagination in such scenario?

Idea 1

Doing static pagination: Pagination is done in browser. This is really simple, but on large result sets it might slow down the device, or even crash the page.

Idea 2

On first load, cache all results and do pagination using cache file (dificulty: medium, also it will not be ok to keep that cached file for too long, since data can change frequently)

Idea 3

No cache, pagination doen on server. Here I have no idea how to do it.

Things to consider

After figuring out the idea behind pagination, I'll also have to add filters and sorting.

Using php, mysql, js.

Answer 1

Let's review your options :

• Option 1 is unpracticable if the result set is very large, the page might become unresponsive and the user prompted to stop the script. This is only true above a few thousands elements. But if on some search result this is the case, it cannot be considered.

• Option 2 - caching solves the problem of latency, but not the problem of volume, a huge dataset in cache is still a huge dataset. I don't really understand how caching simplifies the problem of pagination.

• Option 3 don't have the same drawbacks and only require server processing power. This is why it should be considered.

When working with a database the pagination is handled directly by sending a dynamic limit parameter to the database. Here you only have the option to post-process, so when calling the api in the backend, crop anything above page * pagesize and below (page + 1) * pagesize results. You will probably need to parse the html to do this.

You can later add option 2 which would enable you not to request the API with the same search whenever asked for page 2. You would get the cached request for page 1 instead. Libraries should make this job quite easy.