I have around 200 million new objects coming in, and a 90 day retention policy, so that leaves me with 18 billion records to be stored in the form of key-value pairs.

Key and value both will be a string. It is basically a mapping between a unique identifier for the object in the application to the unique identifier for the object in the actual object storage.

There is an application which loads objects into a Web OS. For each object it loads, it creates a 16 character string key, say DataID. The Web OS itself creates a 40 character string key, say ObjectID. So what I'm trying to do is create a mapping between DataID -> ObjectID for 18 billion objects. I'm don't know the mechanism being used to create the IDs.

I will have to deal with:


I am looking for ideas for an optimal way to implement this. It should be optimized for reads & writes. Space optimization is secondary.

I know Hadoop/NoSQL is one way to go, and probably another solution would be distributed Hash tables, but a few more options would help me decide which is the best solution. A relational database is not an option as we don't have an existing RDBMS in the current environment.

closed as not a real question by Robert Harvey, Jimmy Hoffa, Brian Knoblauch, Dynamic, user40980 Jun 6 '13 at 1:35

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    20 million times 90 is 1.8 billion. Not that it matters much for the question, but is there a factor of ten somewhere you left out? – Karl Bielefeldt Jun 5 '13 at 18:21
  • 8
    Well, even if it is just 1.8B and not 18B you still likely won't be able to hold that all in memory, which means you'd need to be storing it in flat files, probably broken up into chucks. Doing that efficiently is...very hard. With that much data I'd highly suggest getting a database, trying to manage all of that from scratch would be...a lot. – Servy Jun 5 '13 at 18:25
  • 2
    i mean, it should perform reads and writes quickly – Chaos Jun 5 '13 at 18:29
  • 2
    I'm sorry, but I just don't see a question here. There are plenty of resources already out there to help you make your decision, and I don't see how we can add any new value here. A cursory Google Search turns up this: datastax.com/resources/whitepapers/… and this: networkworld.com/news/tech/2012/102212-nosql-263595.html – Robert Harvey Jun 5 '13 at 19:00
  • 4
    If a relational database is not an option because you don't have a RDBMS in your environment, NoSQL isn't because you don't have that in your environment, either. This wheel has been invented many times; don't reinvent your own. – Blrfl Jun 5 '13 at 19:04

Try redis. Its all in memory and dumps data so it can be hot on reset. However you might need to be careful and change the settings if you need to not lose data as it normally waits a second or two before dumping (or did i remember the default settings wrong?).

Use a hash where GUID/6 or 7 bits is the key and the remaining is a field http://redis.io/commands/hmset. Note having more field names make it slower so stick to <=128 as my personal rule of thumb. I recommend having 64 or 32bit but test with the keylength.

The reason I say use a hash is to decrease memory usage. More fields = less pointers (and an increase in CPU time)


Look at these key-value stores: Berkeley DB Java Edition, or JDBM (JDBM3 is the latest), or MapDB (JDBM successor). Tokyo Cabinet is not native Java but has a Java wrapper.

For an overview see http://en.wikipedia.org/wiki/Dbm.

Not the answer you're looking for? Browse other questions tagged or ask your own question.