Is there a best practice for storing multiple pages of JSON responses from an API?

Question

Scenario:

• An API that responds to a request with multiple JSON objects contained in an array.
• The responses are paginated such that each page returns an array of JSON objects.

I deserialize the JSON responses into an object, then append that object onto a List (or some other collection), and then return the collection to the caller. However, is this the best practice? Is there another way, or is it simply dependent on the specific use case?

My efforts to find a best practice thus far have come up short, so it makes me think there is not one, but I thought I would ask this community as well.

Cheers,

Chris

Answer 1

Pagination is often used for a good reason: All elements may be too much to process in a single request, and the caller may be primarily interested in the first few results. If you hide this consideration in your wrapper API, you may create and transfer many more objects than needed. Depending on your specific use case, that may be ok, but there are collections (such as the list of answers and comments for popular Stack Exchange questions) which are much too unwieldy to be handled as a whole.

If you want to wrap the paginated API and hide the pagination in the API you present to your callers, you might consider implementing a streaming API instead, like database interfaces often do for query results (cursors). This enables the caller to sift through you results until a desired object has been found, or until it runs out of patience and just stops using the stream.