I need to design a system where I need to check if a key exists or not. Currently I am already using aerospike for the cause but my use case is slightly different.

I will have a key like this ( A group of 3 integers) --

For eg. 1-105-1133441 222-1000-1891893

I just need to know if this key exists or not in the set. If exists return 1 else set the key and return 0. So that next time the query will return 1

This has to be a tcp service because the checks/set will happen from multiple servers

The point is if a campaign is live the queries will be very dense ( upto 20k/s ). So the key lookup has to be in memory. But after that I still have to retain the set but it has to be on disk , so that memory can be freed up for other campaigns. I do not expect more that 20-30 campaigns to be live.

Whenever the next time an old campaign gets a query again , the server should bring it back to memory and serve the requests from memory. The first query will be slow because it has to kickoff a read from disk but next query onwards will be very fast.

I am using aerospike for now , but there is a problem that aeorspike stores all keys in memory. When it is practically useless to have old campaigns in memory unless they get triggered again

The total number of keys is practially unlimited. But let us consider 50 million keys per campaign and I may need to store 100k campaign data. Of which only < 50 will be live at any given point of time.

  • It sounds like you might benefit from a large cache. Have you considered something like Redis? – Robert Harvey May 21 '17 at 15:20

From your explanations this looks like an use case for a Bloom filter.

A Bloom filter is a space-efficient probabilistic data structure that tells you if a value is:

  • either "possibly in set"
  • "definitely not in set"

Bloom proposed the technique for applications where the amount of source data would require an impractically large amount of memory if "conventional" error-free hashing techniques were applied.

I see you tagged the question [Cassandra] and [Redis]. I'm not familiar with these technologies but a quick search shows that both are somehow using this data structure so see if this is of any help to you.

  • Any bloom filter implementation across machines takes in a lot of memory. Problem is I am storing data for a large amount of time , but only a minuscule amount is required at any given point of time. Dont read everything into memory – Ram May 6 '17 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.