9

The page token approach that you are talking about here is more popularly known as the 'cursor-based-pagination' and depends upon marking the record where the last request left. It can be composed of either a single entity such as an id or composed of more than one entity such as "id+name" hashed together in case the records were requested in a sorted order. ...


8

Note: This is purely speculation, but I had to implement something similar before. Consider a large time-series dataset stored in a big table like HBase. The data doesn't have a constant frequency so you can't make predictions about the density of the data. You want to get page 5000 with page size 25. Since you can't make predictions about the density of ...


6

Get the data into a single index The simplest, no muss, no fuss solution. But then again, why would anything Enterprise be simple? Supply two result sets The best you can do is to provide the first page of each sources answer. If either source runs dry, simply return their set as empty. Don't be tempted to provide more results from the other source, ...


5

It's a tradeoff between speed, memory requirements in the cache, and programming required to keep the cache fresh. A lot depends on your use cases. For example If, like Google search results, most users only ever read page 1, just a few carrying on down to page 2 and beyond, then cacheing may not even be necessary. If users tend to repeatedly sort, search ...


4

There is another kind of pagination that is harder to implements but will do what you want, it usually fits for infinite scrolling and it's more performant. Basically you use the data from the last row as a parameter of your query : SELECT * FROM sales WHERE sale_date < ? ORDER BY sale_date DESC FETCH FIRST 10 ROWS ONLY Where ? will be the value of ...


4

There has been quite a lot written on the subject of api pagination. I think it's fair to say that the naive approach 'get page 5, 20 items per page' doesn't work well with changeable data, filters, sorting etc. ie, if I get 20 items and then apply a filter, i don't have 20 items if I request page 2 of sorted data then I also need to send what I'm sorting ...


4

Isn't it possible to simply cache the thousands of records somewhere to avoid calling the database each time Yes, but it is the exact opposite of being stateless, which REST is. You're trying to go against the main ideology of RESTful services. Can it be done? On a technical level, if you really want, sure. But this is a case of the XY problem. Your ...


3

I don't think that you can come up with anything better here - basically, you're trying to solve an unsolvable math problem, which is joining two sets... without joining them. Technologies like ElasticSearch were constructed to approach this problem by having a single data set to work on. So the way I see it either have to join you data sources by feeding ...


3

There are several possible negatives when considering custom header fields. Browser based testing will be difficult Proxies sometimes remove/mangle headers fields It breaks HTTP cacheing Other developers will not expect it When returning a dynamic list, you will want cacheing disabled, so that shouldn't be a problem. If you never expect to use a browser ...


3

At the level a user-mode process, there are two basic ways in which sequential file IO like playing a movie from start to finish can take place: The process can allocate a buffer, which is typically sized somewhere between a few kilobytes up to a megabyte or so, but rarely any larger than that, and then asks the operating system to fill it up. When it is ...


3

I think that a rest endpoint without path params should always have the potential of returning multiple resources I'm new to REST so I don't know if my view on this is correct or not. It isn't correct. REST doesn't care about spelling, in particular it has no concern for how (or if) the origin server decomposes an identifier to find the mapping to the ...


3

In general, explicit errors are good. But for the specific issue of a pagination limit, silent truncation of the result is probably better. This is not really a fatal error – you will still be returning data. The client can issue multiple requests until they have received as much data as they need. This allows you to document the limit as an upper bound on ...


3

I would recommend not to tamper with ID ranges, this becomes overly complicated and makes unjustified assumptions about the ID value ranges or the order in which a users looks at the blog posts. The posts have unique IDs. Store the IDs of the seen posts per user somewhere, maybe in a table "seen_posts". Then use something along the lines of SELECT ...


2

Fundamentally, your query needs to consider which items have already been seen. Since the position of items in your sort order is not stable (new items can be inserted at the beginning), you need to use a different way to identify the bounds of each "page". Luckily, the relative position of items in your sort order seems to be stable (if an item was ...


2

The only way to know this for sure is to run some performance tests. There are too many factors to make an arbitrary judgment without testing. Many web pages allow you to choose the level of pagination (i.e. the number of records to return) in the UI; do that, and you can easily find out what the sweet spot is with your usual tools.


2

You can use caching pagination, i.e. generate the first few pages, and as the user scrolls through the pages, load more pages in the background. This should give you a good trade-off between initial respond speed and lag between pages.


2

This is performance tuning. You have to try it and see. Try the same way your users will use it when it's deployed. Write code that makes it easy to adjust this setting by only expressing your decision in one place. When in doubt, try it out. Speculation and performance go together like kittens and nuclear bombs.


2

Paging in multi-user environments where data can change between requests for pages is tricky at best, pointless at worst. Say your first request got you a page of 10 items out of a total of 100. Now you do another request for the second page of 10 items, but the number of items has changed to 101, with the new item existing in what would be the first page. ...


2

For pagination, you should be considering the techniques described in RFC 5005. Roughly, each page of results is a separate resource, which itself contains hyperlinks to the previous/next page of results. Then, as usual, the application can just follow the links to navigate from one application state to the next. The actual spelling of the uri doesn't ...


2

I don't consider your example to be a single resource. It returns an array of orders plus helpful paging information. This is typical of operations which return multiple entities. While there is no standard for REST APIs, there are generally accepted conventions. Based on my experience, I would consider this one of them.


2

Step back a bit and look at situation from a cost/benefit perspective: How much is gained by implementing the feature (or how much is actually lost by not implementing it) versus how much does it cost in development and operations to have this feature? Most organizations decide that the benefit of having a stable pagination versus occasional duplicates or ...


2

For an efficient solution you need to be able to put the authorization and pagination constraints into the database query, and have the proper indexes for those aspects. Anything else will potentially overfetch an enormous amount of data. How big an issue this is depends entirely on the scale and characteristics of your data. Can you translate the ...


1

What is best or common way to handle pagination on rest api? RFC 5005 describes link relations for communicating the semantics of paging links to generic clients. It looks about as you expect - each relation includes an identifier of the resource of the appropriate page. It also specifically introduces a distinction between paged feeds (which are lossy) ...


1

If your URI for accessing orders is /api/orders then I would recommend that page 1 of orders be /api/orders/page/1. Then for convenience, if you happen to use the same URI for put, post, and delete requests, you could make it work as if the URI were actually /api/orders (where applicable of course). I would argue against allowing the user to set the number ...


1

There's a pure REST solution to this kind of problem that might work for you, if you are willing to change your UI a bit. Instead of having the user do a GET initially, you change that to a POST. You start running the query and capture the results somewhere. While it's running, you return a URI which points to the results. The UI can then start pulling ...


1

One option (mentioned already really) would be to introduce a caching layer to your system as an additional component. Then you could leave everything the same, except - when you need to query the database, you go through the cache first, if the cache can't fulfill, then the query is passed onto the database. Your cache key would be basically the parameters ...


1

I'v never worked with ASP.Net, but first of all, why not have the data access layer pull only the necessary records? Most relational databases allo you to paginate with something like LIMIT and OFFSET, and I'm assuming Dapper gives you access to these features. Once you have that in place, you could incorporate your page number into your cache key. However,...


1

According to your comments, there's not any specific issue to address. It's matter of establishing rules o methodologies to make your application consistent. That's fine, but the question has many possible and good answers. And in this specific case, mostly opinion based. So, as @CandiedOrange suggest, do some conocept tests with different approaches to get ...


1

Reddit had to solve this too, because content is added rapidly. If you look at their pagination, you'll find URLs like these: ?count=26&before=t3_5jfkqr (link from page 2 to page 1) ?count=50&after=t3_5j99dj (link from page 2 to page 3) Where 5jfkqr and 5j99dj are the first and last items of the current page, respectively and count the current ...


1

Keep in mind that node is non blocking when waiting for io, not if you are processing items within your code. If you are using node to loop through or manipulate large datasets, consider offloading that work to the database or other processes. It also sounds like you are trying to combine database logic (primary keys, rows) with application logic (user ...


Only top voted, non community-wiki answers of a minimum length are eligible