I'm building an API for an ad serving platform that will allow you to request tracker data for ad campaigns. Campaigns often exceed hundreds of millions of requests, which means there will be many terabytes worth of data. Therefore we need to prevent API consumers from requesting too much data at once (such that the request times out), but I'm not sure what the best practice is for doing so is.

Options I've already identified are:

  1. add an extra parameter to the request that indicates which section of the data is desired
  2. truncate the data and somehow tell the client that they need to use more specific filters
  3. respond with HTTP status code 413 (but this appears to be for large request bodies, not responses)
  4. switching to a streaming API (like twitter's streaming APIs)

But my question is, what is the standard practice / proper response for this kind of situation?

Note: DoS attacks aren't much of a concern since this will not be a public API

  • 1
    or make the error part of the API, Sep 3, 2014 at 19:56
  • 2) seems like a bad idea because the client programmer may overlook the flag of "incomplete data". If you fail to provide what the client requests, make it clear that you are not providing it (fail hard, and fail early). I would vote for 3) or better, ratchet suggestion.
    – SJuan76
    Sep 3, 2014 at 19:59
  • recommended reading: Why is asking a question on “best practice” a bad thing?
    – gnat
    Sep 3, 2014 at 20:04
  • @gnat would it be more appropriate to just ask what solutions others have successfully implemented?
    – Griffin
    Sep 3, 2014 at 20:53
  • unlikely, as this would make it a list question with known issues. Why don't you just copy the question from title? "What is the proper response etc"
    – gnat
    Sep 3, 2014 at 21:00

3 Answers 3


Return the harshest, unfriendliest result possible in the event of a malformed request (one that returns more data than your metering allows is malformed). I suggest returning a 4** error code. Then, also provide paging parameters, so that users may request pages. oData has this feature, for instance. Do not truncate the data silently, under any circumstances.

Consulting with customers is a bad idea. They are going to tell you to do whatever possible to minimize errors, which is a bad engineering approach. This is your decision, take it by the horns and do the right thing.

An example of a paginated api is oData:


  • +1. 412, 413, 416, 417 are correct responses.
    – Residuum
    Sep 3, 2014 at 21:28
  • can you give an example API that batches / paginates the results?
    – Griffin
    Sep 3, 2014 at 22:09
  • @Griffin edited to reflect an example Sep 4, 2014 at 19:05

To expand on what @joshin4colours said, I think you have a false dichotomy (trichotomy?). Why not provide all three solutions? Maybe the default is to return a 413 but with other flags you can either get some of what you want with an embedded error in the data and/or provide a way of batching the data.

It really depends on what your specific customer/consumer of the API expects and how they want to use your API. Are they ever going to want a 413? Should the default response include some data and indicate how much more there is? Maybe. You could also put yourself in the client's shoes and think about what they would want, i.e. what would be useful to them.

What I've usually done is give the first batch of data with an idea of how much more there is. Returning a 413 isn't very friendly, but maybe that's want you want in some cases. From what I've experienced, usually there's a default batch size but people can ask for a certain batch size up to some limit.

Also, you could consider aggregation or sampling to reduce batch size. For example, I want 50,000 results as a random sample of 5,000,000 matching records. There are different ways to slice and dice depending on how statistically significant you want your results to be.

  • right, consulting the actual customer(s) is always a good idea. In the mean time I'd like to explore what solutions have worked for others.
    – Griffin
    Sep 3, 2014 at 21:01

Not sure about a best practice, but in our case we do have parameters in our API that are set to some kind of maximum value (think Integer.MAX_VALUE from Java). These parameters are often not available to the UI/client-side of the application, only to server-side calls.

Basically, the approach would be to set a maximum on records returned by your request. Seems to work well, particularly when data doesn't need to be organized or paginated in any way.

If a client (human or otherwise) need more than this maximum, you may want to consider increasing it or batching your data somehow.

  • 1
    and at the very least document the maxes when they leak through the abstraction Sep 3, 2014 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.