API pagination with external or centralised authorization

Question

I am building a REST API which would power a front end as well as other 3rd party apps and hence I want it to be as "standard" as possible. Right now, I am trying to stick to HATEOAS. The only place I am struggling is pagination.

The authorization layer of our application is centralised. Multiple other apps use the centralised auth service and so my API needs to use the same. This gives rise to obvious problems in pagination, namely:

1. How to restrict the number of entries to a required number?
2. How to return entries of any valid page?
3. How to return the total number of pages?

Right now, I am using an ad-hoc solution that fetches all records from the database, according to the API filters, and then the authorization layer, filters the unauthorized records and then another layer (lets call it the "Pagination Layer") filters according to the page parameters.

This works for now as our dataset is relatively less but I don't think this will scale well. What are my options?

P.S. There are a few things I have thought about but have no idea how good of an idea they are:

1. The frontend can be switched into a lazy loading mechanism so that returning exactly a certain number of entries is no longer mandatory. The frontend is take care of querying the next page if required. But, this will give a bad experience to 3rd party devs using the API.
2. The business layer gets only the number of records as in the page and the pagination layer decides if more queries are required to get more data. This looks like a bad idea in many ways as this won't solve getting a particular page.

Answer 1

For an efficient solution you need to be able to put the authorization and pagination constraints into the database query, and have the proper indexes for those aspects. Anything else will potentially overfetch an enormous amount of data. How big an issue this is depends entirely on the scale and characteristics of your data.

Can you translate the information you get from the authorization layer into a filter on the database query? Ideally something like getting the information "User A can access Projects X,Y and Z" and translate that into filters on your query.

If you cannot do that and have to pass every single result to it to know whether it is visible, you will always have some pathologically slow scenarios. For example if you have 1 million items, and your current user is allowed to view 10 of them, you might have to push the entire million items through the authorization layer just to get 10 results. How big of a problem this is depends heavily on the specifics of your application.

If you cannot push all these concerns to the datbase, which I assume is the case from your description, I think something like your solution 2 is the only reasonable way to handle this. You essentially need an internal pagination layer that fetches a bunch of results, passes them through the authorization layer and provides them to the rest of your application. Your externally-visible pagination layer then needs to request internally pages results until it has enough to fulfill the request.

This has the issue I mentioned above with potentially pathological queries in terms of performance, but I see no way to avoid that with these restrictions. There is also no fundamental issue with querying specific pages in this way, it's just expensive as you have to also query all pages before. But that is a general issue with pagination unless you can use advanced methods like keyset pagination.

If it is possible, you can also simply avoid providing the option to query specific pages. So you'd only provide a "next" link in each paginated response. This gives you the largest flexibility in designing your pagination, but obviously restricts what the client can do.