I am working on problem where a call to reduce a counter will come to a service and if counter is greater zero then call should be able to reduce it else fail.
Pretty straightforward? huh!
For a request get the counter value, reduce and put it back
Well it becomes interesting with below Constraints:
- Request is sandboxed: So request can come to any host, each request creates a new thread and dies after returning the response. (So no batch update possible out of the box, in other words can't update counter with -10 on behalf of 10 requests if each request wanted to do -1)
- Maximize success rate of parallel requests for same counter update
- Minimize latency impact due to your solution (<700ms)
- Counter is stored in some data store (lets say DynamoDB, may not be the right data store for accessing same key with high rate as it causes hot partition and increasing the throughput just to support this weird call pattern is not acceptable)
Whats the problem exactly, you ask?
Accessing same record many times tend to create hot partition scenario where most of underneath data store starts throttling you as your requests/access pattern seems sort of attack. (don't suggest keeping high throughput to support the pattern, not acceptable!)
Directly processing a request used to work when there is no contention or say not many parallel request updating the same counter. Now most of the requests (99%) will fail due to lock/conditional fail cases and retries will take hell lot of time for all of them to succeed. (I am ok few requests fail ~10%)
About failures: "failure due to counter reaching 0" is not retryable while "failure due to lock/conditional fail cases" is retryable.
Aim is to maximize the success rate of parallel request as much as possible.
I am not limited or restricted with particular data model or store. That means you can come up with any data model which help you crack the problem is efficient way and choose any data store you believe is right for such use-case.
I have a fairly good solution(using randomness) which I can talk about later. (Not putting it upfront in order to keep problem open and interesting to be solved rather than discussing a single solution)
Wanted to collect thoughts here, how you will approach it!