In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently.

The worst case scenario is a 5-10MB string. I'll have thousands of instances of my object and I'm worried that storing all those strings in the filesystem would not be optimal, but I can't think of a better solution.

Can anyone suggest an alternative? Maybe a key-value store? In this case, which one? Any other thoughts?

  • What do you mean with 'instance of an object"? An object is already an instance (of a class). Are the 'very long ASCII string's needing to be added to multiple objects? Or registered just once? What is the append? Appended to what? A string that is already stored on the object or do you mean 'inserted' or 'stored' ? – Joppe Dec 8 '12 at 0:59
  • The string is already built and is mapped to an object on a one-to-one relation. By appending I mean the operation of inserting one more character at the end of the string. – Thiago Moraes Dec 8 '12 at 16:05

Appending a few characters to a 10mb string sounds like a poor design choice. Back up and consider what you are actually doing, without the assumption/conclusion that the object in question is a string. Maybe some (infrequently used) view of the object needs to be a string, but that is another matter.

| improve this answer | |
  • It is indeed the case. The string is a view that could be generated on the fly based on the actual representation of the data, that will be stored on a relational database. But retireving 5 million rows to get a 5 million characters and then generate a string appeared to be an even poorer choice for me. I would happily consider alternatives, but by myself I'm just not imagining any. – Thiago Moraes Dec 8 '12 at 16:10
  • Consider saving all the appends as separate items, but constructing the canonical string and saving it when the full string is first required (or when the number of appends reaches some threshold) – ddyer Dec 8 '12 at 17:56

Insert them as separate items/rows. Append them together on read.

This can be done in relational DB or simple file.

| improve this answer | |

Heres how I would solve this:

I would store the data as a series of blocks within 2GB files. Each block needs the string ID, the block sequence for that string, file ID and file offset of the previous block and around 2k of data space for writing the string chars into. When the data space for that block is filled, then you assign the next unused block to continue the data stream and post back to the previous block.

For each object string, you need to keep in memory the block sequence, the file handle, file offset, and remaining bytes that can be added to this block.

When you need to recreate the string to read, you know how big the string is, based on block sequence number, and then have to read in the data from each of the blocks.

Based on 1000's of objects with 2 writes a second per object, the disk infrastructure needs to be considered.

| improve this answer | |
  • My memory footprint will be very big, unfortunatelly, doing it in-memory is not an option. Reading the other answers I'm trying to come up with a different model for my problem, maybe storing slices of each string in a row on a relational database and then retrieving all of then, for example, but haven't concluded anything yet. – Thiago Moraes Dec 8 '12 at 17:21
  • answer updated based on the memory footprint info – Michael Shaw Dec 9 '12 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.