I'm building an API for an ad serving platform that will allow you to request tracker data for ad campaigns. Campaigns often exceed hundreds of millions of requests, which means there will be many terabytes worth of data. Therefore we need to prevent API consumers from requesting too much data at once (such that the request times out), but I'm not sure what the best practice is for doing so is.
Options I've already identified are:
- add an extra parameter to the request that indicates which section of the data is desired
- truncate the data and somehow tell the client that they need to use more specific filters
- respond with HTTP status code 413 (but this appears to be for large request bodies, not responses)
- switching to a streaming API (like twitter's streaming APIs)
But my question is, what is the standard practice / proper response for this kind of situation?
Note: DoS attacks aren't much of a concern since this will not be a public API