1

Given the following assumptions:

  • The amount of available data storage is highly limited
  • All data is international
  • As much storage as possible must be preserved since it will be used for something else

Is there any widely accepted standard or best practice which defines how much storage should be allocated for those items?

Note: I'm not looking for assumptions but for facts like "97.5% of all names fit into 70 bytes using UTF-8" and "An international address must at least contain field1, field2 and field3 of size1, size2 and size3".

Edit: I'm looking for the fields/lengths to use in the user interface. Not a technique to store it efficiently.

4
  • If all data is international, many chances are that their UTF-8 representations will be wider than 2 bytes, so you might actually end up saving space by choosing UTF-16 encoding. You should check this.
    – Mike Nakis
    Commented Jan 1, 2012 at 16:20
  • It may help you to look at postal authority standards in your specific geography/country.
    – NoChance
    Commented Jan 1, 2012 at 21:23
  • @Emmad i'm looking for something that works across all countries
    – mibollma
    Commented Jan 2, 2012 at 13:11
  • stackoverflow.com/questions/20958/…
    – JeffO
    Commented Jan 2, 2012 at 15:07

2 Answers 2

3

Coming at it from a slightly different perspective, since your goal is to store things as compactly as possible, avoid using fixed width fields.

To pull from your own example:

97.5% of all names fit into 70 bytes using UTF-8

You shouldn't have a fixed 70 byte allocation for the name.

Instead, have a fixed allocation for the entire record - say, 200 bytes - then store all the data in that one record with a delimiter between fields.

For example:
Title/Given/Middle/Family/KnownAs/Born
Mr/William/Henry/Gates/1955-10-28

This is a classic space/time tradeoff as searching such a structure will be slower.

One key advantage of this structure is that it allows for some fields to be unusually long without compromising storage. So, if someone's street address happened to require 70 characters, or if their full name took 90, you can accomodate this as long as the same person didn't have both.

2
  • 1
    Why stop at dynamic sizing for record elements. It is entirely possible to have variable sized records (e.g. Just use a different delmiter for end of record). This is a trivial problme if speed performance is not a major consideration, and gets interesting while being entirely solvable even if it is. Theres not enough informtion in the question to suggest what solution to use though.
    – mattnz
    Commented Jan 2, 2012 at 7:21
  • I think this doesn't translate well into the user interface though: "Error shorten any field". I'm looking for the size limits and fields in the user interface. Once they are defined it is true that additional space might be saved using your approach.
    – mibollma
    Commented Jan 2, 2012 at 13:21
2

If the space is limited, then the overhead of skipping elements one-by-one wouldn't be so important.

Thus, you can construct a tree, made of lists of length-prefixed elements. You can encode the lengths as single bytes, and for records meant to be larger than what this allows, you can shift the value to the left (that wouldn't waste because it's compensated by the smaller field).

Also, if you want to simplify your code further, it might make sense to embed a flag in the size field indicating if the record is raw or a list. That would make trivial to parse the whole data set (either incrementally or on a single pass).

That would be much more efficient than the fixed-length statistical approach.

Plus, you might want to consider applying some form of compression to the whole thing to save some more space.

7
  • This is already too far ahead. First i'm looking for fields and lengths suitable to work across all countries. Once they are defined we could talk about an efficient way to store them... although i would probably use JSON, Protocol Buffer or MessagePack anyway.
    – mibollma
    Commented Jan 2, 2012 at 13:28
  • But why would you want to limit this on the UI? Commented Jan 2, 2012 at 18:07
  • Assuming it is unlimited. How do you suggest to tell the user if the sum of his inputs exceeds the combined storage limit? I can't think of any user friendly way.
    – mibollma
    Commented Jan 2, 2012 at 19:31
  • Why? it's very simple, anyone can interpret something like "not enough space left, try cutting down at least N characters". It doesn't needs to be more specific than that... Commented Jan 2, 2012 at 20:43
  • 1
    Or, a much better approach, put a small counter somewhere in the border of the screen. It can be a small color coded counter, even if it doesn't has any caption, everyone will deduce the purpose as soon as they start writing. Commented Jan 2, 2012 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.