3

So, I have a bunch of calls that are all being generated with a UUID1 throughout the day. At the end of the call the call is processed and some metrics around that call are generated and stored in Rethinkdb/Cassandra.

Each call will generate something that looks like this

[{
   "company": "foo company",
   "campaign": "bar campaign",
   "hash": "",
   "stat": "talk-time",
   "date": 1467356399,
   "value": 176
 },
 {
   "company": "foo company",
   "campaign": "bar campaign",
   "hash": "",
   "stat": "sale",
   "date": 1467356399,
   "value": 1
 },
 {
   "company": "foo company",
   "campaign": "bar campaign",
   "hash": "",
   "stat": "call-back",
   "date": 1467356399,
   "value": 0
 },
 ...
 ]

I need these stats to be unique within the database. My current solution is to take the UUID of the Call that is stored in Postgres and add it to a string that will look something like this for the first stat uuid1_foo-company_bar-campaign_talk-time_1467356399 and hash it using a SHA-512. Currently, I am using that hash as the ID on Rethinkdb to gain uniqueness.

The reason these need to be unique and reproducible is because sometimes we have to go back and reprocess all the calls for a given day and we need to ensure that stats generated the first time they were processed we kept and not duplicated. If they are duplicated all our reporting would be incorrect.

Is there a better way using these tools to generate unique stats for a call that can be reprocessed later without inserting duplicate values?

Also, it seems that Rethink has a max length of 127 for the primary ID where SHA-512's are 128 characters long, hence rethinking this design.

  • What's the point of the SHA-512? Since the string incorporate all the other components, they should already be unique (right)? If there are duplicates, then you'll get the duplicates in the SHA-512, too. But if there aren't duplicates, then the SHA-512 is adding a very small chance of creating some. Leaving the string in a "human readable" form might also make some kinds of debugging easier, too. – Joshua Taylor Jul 5 '16 at 18:21
  • @JoshuaTaylor Changing it from readable to the SHA was just to uniform the length of the string. Essentially the human readable should be unique within the application. It would be easier to debug in the readable form as well, yes. The only piece that is not incorporated is the value that is left to be just the only real piece of data. – Jared Mackey Jul 5 '16 at 18:25
  • There are ways of getting a uniform length string that wouldn't add the possibility of duplicates, though. E.g., since the UUID never contains spaces, you could left-pad with spaces until the string is whatever length. (That's just one option, of course.) That won't introduce duplicates like the SHA-512 could, and the string stays human readable. – Joshua Taylor Jul 5 '16 at 18:27
  • Why not just have company name and the other values as separate columns in the postgres table, and uuid as another column, with a unique constraint covering all the columns? – Daenyth Jul 5 '16 at 18:28
  • @JoshuaTaylor I do like that idea and it would be a good possibility if I were able to change the max length of the ID in rethink. The examples above are rather short examples. – Jared Mackey Jul 5 '16 at 18:29
1

I see two possibilities for creating a new key:

  1. Generate A UUID5, which is based on the SHA-1 hash of a namespace identifier (which is a UUID) and a name (which is a string). Use your original UUID and a unique string combination within your record, or

  2. Generate a SHA512 hash of your entire record, encode it to a base64 representation, and append the resulting 8 characters to the end of your original UUID.

  • I went with option 1. I ended up having rethink create a UUID5 from the string that I was previously hashing and then concatenating that UUID5 to a UUID1 of the parent object. – Jared Mackey Jul 11 '16 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.