How to best protect a public API from unwanted requests

Question

I’ll try my best to explain, but for the closest context I could think of, imagine that I am building an analytics platform that allows paying users to sign up, place a tracking script on their website and track their website visitors’ data. This would mean anyone who visits their site would need to be able to make an API request (without them even necessarily knowing - done via the tracking script) to send the analytics data to the database. E.g. users may install a script in the head like this:

<script data-domain="somedomain.com" src="path/to/tracking/script.js?id=SOMETHING" defer></script>

I also wouldn’t want users to have to ever update this script.

My scenario is somewhat similar, but to summarise, I worry that anyone (e.g. a hacker) would be able to insert a load of junk data into the database (since they could make API requests that require no auth).

What could be done to prevent this, but not prevent the script from tracking all users? I was thinking along these lines:

1. IP rate throttling to cap the no. of requests someone can make
2. Auth tokens send in request headers (but they expire, unless they could be updated automatically some way? From another script? But couldn’t a hacker also access this anyway?)
3. Ensure that the API endpoint used by the tracking script is designed to only accept requests from the tracking script, and not directly from other sources (e.g. how? via CORS headers?)

But presumably authenticating requests is not an option since it would prevent tracking everyone else.

I could be wrong in my assumptions here, but any ideas would be greatly appreciated. It just feels weird, since anyone would be able to insert a load of junk data into the database. It may not be the signed-up user who misuses the API, but instead a random hacker visiting their site

Answer 1

Your answer is basically this

IP rate throttling to cap the no. of requests someone can make

But you can obviously get a bit more sophisticated than just simple IP rate throttling - you can identify anomalous patterns in the requests (using all of the source, the rate/pattern of requests and the content of the requests themselves) and then discard those requests one way or another.

Note that in some cases in may be advantageous to allow the attackers to think they are writing to your database but in fact you are just discarding the data (or keeping it somewhere else to improve your pattern matching for defending against future attacks).

Answer 2

There seem to be three threats to protect against here:

1. Invalid data.
2. Metric abuse.
3. DOS/DDOS

1. Sanitize the received data.

Presumably valid customers/users send a known set of fields.

Timestamps, ids, ...

You can add filters such that only valid values will be accepted. For strings you can probably filter on length and possibly code page (if you don't need I18N support).

2. Metric Abuse

You can make it more difficulty for someone to game/abuse your statistics by adding a token to your script that you rotate on roughly the same timeline as the cache-control headers expire your script.

And/Or

Rotate the tokens that the legitimate customer websites are using - you will need to provide an authenticate API for them to fetch new tokens. How often you rotate the tokens determines how often the "abuser" will have to grab a new token from the customer website.

But neither of these are a true defense they just make it more difficult for someone to game the system.

The simplest defense is probably a statistical analysis. When someone writes a bot to game a system it is difficult to make it look identical to real user traffic. For example the number of requests per IP will increase where as organic traffic growth will typically increase the number of IP's.

3. DOS/DDOS

There is a limited amount you can do yourself - basically find a way to reject the request without needing to spend significant processing resource on your side.

For a significant attack you are probably going to need to take your origin servers private and then use a CDN to provide the DDOS protection for you.