Proper Response to HTTP Request When Too Much Data is Requested

Question

I'm building an API for an ad serving platform that will allow you to request tracker data for ad campaigns. Campaigns often exceed hundreds of millions of requests, which means there will be many terabytes worth of data. Therefore we need to prevent API consumers from requesting too much data at once (such that the request times out), but I'm not sure what the best practice is for doing so is.

Options I've already identified are:

1. add an extra parameter to the request that indicates which section of the data is desired
2. truncate the data and somehow tell the client that they need to use more specific filters
3. respond with HTTP status code 413 (but this appears to be for large request bodies, not responses)
4. switching to a streaming API (like twitter's streaming APIs)

But my question is, what is the standard practice / proper response for this kind of situation?

Note: DoS attacks aren't much of a concern since this will not be a public API

Answer 1

Return the harshest, unfriendliest result possible in the event of a malformed request (one that returns more data than your metering allows is malformed). I suggest returning a 4** error code. Then, also provide paging parameters, so that users may request pages. oData has this feature, for instance. Do not truncate the data silently, under any circumstances.

Consulting with customers is a bad idea. They are going to tell you to do whatever possible to minimize errors, which is a bad engineering approach. This is your decision, take it by the horns and do the right thing.

An example of a paginated api is oData:

http://www.odata.org/documentation/odata-version-2-0/uri-conventions/

Answer 2

To expand on what @joshin4colours said, I think you have a false dichotomy (trichotomy?). Why not provide all three solutions? Maybe the default is to return a 413 but with other flags you can either get some of what you want with an embedded error in the data and/or provide a way of batching the data.

It really depends on what your specific customer/consumer of the API expects and how they want to use your API. Are they ever going to want a 413? Should the default response include some data and indicate how much more there is? Maybe. You could also put yourself in the client's shoes and think about what they would want, i.e. what would be useful to them.

What I've usually done is give the first batch of data with an idea of how much more there is. Returning a 413 isn't very friendly, but maybe that's want you want in some cases. From what I've experienced, usually there's a default batch size but people can ask for a certain batch size up to some limit.

Also, you could consider aggregation or sampling to reduce batch size. For example, I want 50,000 results as a random sample of 5,000,000 matching records. There are different ways to slice and dice depending on how statistically significant you want your results to be.

Answer 3

Not sure about a best practice, but in our case we do have parameters in our API that are set to some kind of maximum value (think Integer.MAX_VALUE from Java). These parameters are often not available to the UI/client-side of the application, only to server-side calls.

Basically, the approach would be to set a maximum on records returned by your request. Seems to work well, particularly when data doesn't need to be organized or paginated in any way.

If a client (human or otherwise) need more than this maximum, you may want to consider increasing it or batching your data somehow.