I need to design a system where I need to check if a key exists or not. Currently I am already using aerospike for the cause but my use case is slightly different.
I will have a key like this ( A group of 3 integers) --
For eg. 1-105-1133441 222-1000-1891893
I just need to know if this key exists or not in the set. If exists return 1 else set the key and return 0. So that next time the query will return 1
This has to be a tcp service because the checks/set will happen from multiple servers
The point is if a campaign is live the queries will be very dense ( upto 20k/s ). So the key lookup has to be in memory. But after that I still have to retain the set but it has to be on disk , so that memory can be freed up for other campaigns. I do not expect more that 20-30 campaigns to be live.
Whenever the next time an old campaign gets a query again , the server should bring it back to memory and serve the requests from memory. The first query will be slow because it has to kickoff a read from disk but next query onwards will be very fast.
I am using aerospike for now , but there is a problem that aeorspike stores all keys in memory. When it is practically useless to have old campaigns in memory unless they get triggered again
The total number of keys is practially unlimited. But let us consider 50 million keys per campaign and I may need to store 100k campaign data. Of which only < 50 will be live at any given point of time.