2

i have a case where user can filter and download multiple pdf documents merged into single file.

The client should display an error message when no document's meets the filter, and ask for confirmation when there are many documents because it might take a while to process them (let's say the confirmation should trigger at 10 documents).

For now i've implemented it as described below, but it just seems off.

Request: GET /foo?limit=10

The results are:

  • Everything is correct: status 200 OK, the body would contain download link.
  • No documents found that are matching the filter: status 404 NOT FOUND, empty body
  • Found more than 10 (requested limit) documents that are matching the filter: honestly no idea what the status should be, for now i am returning 429 TOO_MANY_REQUESTS but it's not right. The body contain the amount of documents so the user can either abort the operation or create new request with higher limit.

Im sure that im missing something important, i have not enough experience in the field. What is the right way of handling this?

Other solutions that i've been thinking about:

  • using custom http codes, but i've come to a conclusion that it's probably a bad idea (even if they are free now, i don't know whether they might be used as official http codes in a year or ten years from now, besides, it feels like an overkill).
  • each of these situations resulting in 200 OK and give more info in the request body
1
  • Is the processing of the merged pdf documents syncronous or asyncronous (ie does it happen immediately or does the client request it and then have to wait for a while before it can actually download it from the link?) If it happens immediately then just make GET /foo return the merged PDFs, the second download link seems unnecessary Oct 13, 2020 at 11:02

5 Answers 5

10

The client should display an error message when no document meets the filter, and ask for confirmation when there are many documents because it might take a while to process them (let's say the confirmation should trigger at 10 documents).

Both cases are 200 OK.

  • OK, Foos resource exists, but I didn't find the subset you are looking for
  • OK, Foos resource exists, but the subset you are looking for is larger than expected.

each of these situations resulting in 200 OK and give more info in the request body.

Absolutely

200 OK
{ limit:100, found:101, link:"<link>" }

In both cases (limit exceed or not), there will be a link to follow. Isn't it?

200 OK
{ limit:100, found:0, link:null }

If no document is processable, then there's no link to show.

In all the above cases the service/API did the job, it worked, the client sent the right input to the right endpoint, there's nothing wrong with that. Don't make things confusing. 404 is misleading. The developer has to guess whether it's pointing to the right URI or not. 400 is too, the developer has to guess whether it's missing any important argument.

Use HTTP status codes to communicate with the HTTP client instead of the web application.

We use HTTP status codes to make the HTTP client behave in one way or another. For example, if no confirmation is required we can send 302 Redirect and put the download link into the response headers and let the protocol to do its magic. If we want the HTTP client to wait before issuing the redirection, we send the header Retry-After along with the 3xx status code. As you may guess, our web app doesn't need to know and understand the semantics of Retry-After the HTTP client will.

3
  • My impression is that the output should be a concatenation of all files. So with limit = 100, found = 101 the link should be null and the caller decides whether they want to try again with a larger limit or not.
    – gnasher729
    Oct 5, 2020 at 15:57
  • Could be. Note that, the link could the same /foto?limit=[N] where N is equals to found. No need to be the link to the binary.
    – Laiv
    Oct 5, 2020 at 22:31
  • The thing is, i don't want to process unnecessary large amount of files because it is very expensive - it's calling external API's and generating large amount of files can take a while. It could be redesigned not to generate the file unless there is download attempt though. Still, your solution seems to be the cleanest one, i think that eventually, in the next refactor i will give it a go. Oct 6, 2020 at 9:47
6

As an API consumer I would think that adding ?limit=10 would return the first 10 matching results, not throw an error if there are more than 10 results.

You could consider changing from the limit to an Expect HTTP request header.

Then if there are more results then expected, return a 417

2
  • Seems like a good idea, much more intuitive. Is 404 when there are no documents matching the filter okay though? Oct 5, 2020 at 9:32
  • 1
    Yes, a 404 is the best choice when no documents are found.
    – Rik D
    Oct 5, 2020 at 10:00
3

If I am querying an endpoint that returns a list of items, I would not expect an error code in any of these cases. If I request a list of items matching filter X, and there are none, what I expect as return is an empty list. My query is valid, the endpoint exists, the filter exists and I passed an acceptable value to the filter. So my request is well-formed, and the answer in that case is simply that there are no items matching that request. That is a valid answer, not an error.

As for the "too many documents" part. The solution in a typical case would be to properly paginate the results. So you would return 10 results in this case, with metadata about pagination that indicates whether there are more results available. Then the client can use that metadata to prompt the user.

In your case where you return a merged document this isn't possible. I would separate the endpoints for this purpose entirely, if that works for your application. So the filtering endpoint would return all results (ideally paginated) so that the client knows which documents and how many are in that selection. The second endpoint would take a list of document IDs, and return a merged PDF. The part about warning when too many documents are selected is a concern of the client, not of the web API. Of course you also should set a limit on the server, but that would be a global, not overridable limit to prevent DOSing the system or overloading it accidentally.

I'm making some assumptions here about your case, but from what I read I would find your example API very confusing and the behaviour unexpected.

3
  • My impression is that pagination is not wanted, that the caller wants either a PDF file containing the concatenation of all documents, or nothing if that is too expensive. Because processing paginated results is very non-trivial.
    – gnasher729
    Oct 7, 2020 at 5:54
  • @gnasher729 I missed the "merged into a single file" when I read this initially. I think the real solution would be to decouple the filtering files from the merging files API call, so that you always merge an explicit list of IDs, the current design it too coupled to the desired client behaviour. I added that to my answer. Oct 7, 2020 at 6:52
  • That sounds quite reasonable. I would assume that a list of file ids is never too big to return in one go. (And if it is, then surely the merged files are much much much too large).
    – gnasher729
    Oct 8, 2020 at 13:08
0

The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting").

RFC 6568 indicates that status 429 should be reserved for rate limiting only. That makes it a very poor status code for indicating too many results. Instead, consider adding an offset parameter to go with the limit. When more than limit documents are available, your API should include a URL to the next page of results (/foo?limit=10&offset=10). If you must decline the request when the limit is too low, I'd recommend a very standard 400 Bad Request, with more detailed information in the request body.

And while a 404 may be used to indicate there are no documents matching the filter, your API should provide a more detailed status in the response body. After all, the status does not make clear why your API didn't return any results; perhaps the client misspelled the URL, and tried to search on /goo instead.

0

The client should display an error message when no document's meets the filter, and ask for confirmation when there are many documents because it might take a while to process them (let's say the confirmation should trigger at 10 documents).

If the final decision rests with the client anyway, who can choose to set the limit to any arbitrary value they want, it'd be easier for you to just always return the link and the information on how many files this link contains.

Instead of sending a confirmation to you, the client can simply look at the returned file count, evaluate it for themselves and decided whether to access the link or not.

This removes the need for the limit input parameter:

Request: GET /foo

And you simply return your findings to the client:

200 OK
{ files: 250, link:"..." }

What I find counterintuitive about your intended approach is that you put the responsibility of deciding the limit with the client, but your API forces the client to use the decision logic that your API has chosen to implement.

Who says they always care to only process a certain number of files? Even if that is the case today, who says that it's going to be the case tomorrow?

If tomorrow you have a client who only wishes to process a minimum amount of files (instead of a maximum), then you'd need to redevelop your logic and start accounting for all possible decisions that a client could want to make.

Rather than forcing a filter that does not impact your own service, simply return the requested information and let the client decide what to do with the provided information. By returning the file count to them, you leave the decision making process to them, so you don't have to develop every possible evaluation yourself.

This gives you the best of both worlds:

  • Less complexity for you to implement (no confirmation logic etc)
  • The client can evaluate their own decisions without needing you to update your API. If the client changes how they evaluate things, they don't need to wait for you to update your API logic.

Just to finish my thought: if the file count does impact the performance of your API, e.g. because it's your API who performs the expensive task of processing the files, then I would expect your API to enforce a reasonable file limit.

It makes no sense to want to defend against something (overloading your server) and then not regulating it yourself and giving the client full control over whether to do so or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.