I have an api where visitor can send an email through subscription:
/api/subscribe
To prevent massive load due to public exposure, how can I secure this endpoint? Do I have to use database or can I do it without that with some kind of caching, inmemory etc which releases ever 10 minutes etc?
You can use Proof of Work to enforce rate limiting without needing to remember IP addresses.
With Proof of Work, you require that the client do a computationally expensive function to generate a proof that you can verify cheaply. One of the most common PoW is partial hash inversion, in which that you require that every API submission is attached with a hash of proof+request, for which the hash must have a predefined prefix (usually zeros) of a certain length. For the client, calculating the proof requires testing hundreds or thousands of potential proofs, until they stumbled upon one that have the required number of zeros; the goal should be to require typical clients to expend several seconds or minutes of computational power for each messages sent. But for the server, verifying the proof involves only one hash calculation. This asymmetry means it's much more expensive for the client to generate valid request, than for the server to reject invalid ones.
To prevent replay attack, you can require that requests must contain a timestamp or a counter. Your server can check to reject requests containing timestamps that are too old or which contains a counter value that had already been used.
I have done this, so I know how it is optimally done. The idea is to use a hash function such as SipHash to calculate a hash value for the IP address. Then you use a token bucket algorithm for each hash bucket: have e.g. 100 initial tokens in each hash bucket, add 10 tokens per second up to a maximum of 100 tokens, and remove one token every time you get a request, or else if there are no tokens, reject the request. This would allow 10 requests per second with a maximum burst size of 100.
Theoretically, it is possible that two IP addresses hash to the same bucket, but that is not a problem in this use case, if you have enough hash buckets.
As for updating the buckets, you can do them using batch timers. E.g. for 131072 buckets, you could update e.g. 4096 buckets per each timer and then have 32 timers evenly expiring within a second. So, at 1/32 seconds, you update the first 4096 buckets, at 2/32 seconds, you update the next 4096 buckets, etc. The data structure for maintaining timers is optimally a priority queue such as a binary heap.
When implemented this way, if somebody floods your system by numerous forged source IP addresses, your memory doesn't get filled.
The memory used by this approach uses 8, 16 or 32 bits per each hash bucket if you use an integer array. The integer size comes from your requirements: e.g. 8 bits cannot support more than burst sizes of 255. Similarly, 16 bits allow burst sizes of at most 65535. So, e.g. 8 bits or 1 byte per bucket and 131072 buckets takes 128 kilobytes of memory. Nowhere close to being a problem. A good machine has at least 2 GB of memory, meaning over 15 000 times the amount you require for this system.
You need to consider memory bandwidth as well: if each bucket is updated once per second, the bandwidth required is 128 KB/s. Good computers support over 5 GB/s read+write bandwidth, or over 40 000 times what this proposal of mine uses.
Do save the cache into RAM. Don't use a database or disk file for it. If your system crashes, well, then you just initialize all buckets to the initial value.
By public exposure, you mean that the API itself doesn't require any authentication? You may reconsider the choice of making it public in the first place. Anything which costs you resources and additionally could get you in trouble (sending too many e-mails would get you in trouble with both spam filters and people receiving those e-mails) should not be publicly available, but restricted to the registered users that you are able to ban (or, better, make each request paid).
If, for some reason, you're unable to require authentication, there is not much you can do, unfortunately. You may search for DOS and DDOS attacks; you'll find a lot of resources, but none will insure you with certainty that the API would be used by wise persons for legit purposes only.
As for the blocking per IP, this is the most basic, the least effective and also the most problematic protection against DOS and DDOS. Its major problems are:
The fact that the same IP address can be used by multiple persons. In the company I work in, we are probably several thousand sharing the same public IP address (or a small range of IP addresses).
This caused a few issues with some websites, including Stack Exchange, which claimed several times that it received too many requests from the IP address, and blocked us. Well, indeed, hundreds of developers working at the same time are generating a lot of requests.
In a previous company, it's Google Search which occasionally blocked us, requiring to fill a CAPTCHA. Here too, hundreds of persons working at the same time may do a lot of Google searches every minute.
The fact that a DDOS attack is specifically performed from multiple IP addresses. You may get hundreds or thousands of requests coming each one from a different IP address, all arriving at your servers at the same time. And even if you try to block those addresses (which may belong to persons who had no intention nor skills to do any harm to your server and who doesn't even know your website exists), the attacker can relatively easy switch to other machines.
I ended using WebApiThrottle which you can use to limit request amount on selected endpoint.
I am using AWS, specific API Gateway to expose my api, API Gateway lets improve rate limit (they call quota) thus, the users cant make x req/seg. For me was the easily way to do that. http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html
