2

We have an app that receives a web service request, processes it and sends it back to our client by another web service call. There is a unique field in the request, a tracking Id, which currently follow the pattern [A-Z][A-Z][-][0-9][0-9][0-9][0-9][0-9][0-9]. It is better explainable with a sample in that format, let's say we have the tracking id "AB-123456" where "AB" is always alphanumeric, "123456" is currently numeric, and "-" is the separator. We have a length and format restriction in place on our client-side.

Now, we want to update the format to "AB-123456-1" to meet a new business requirement (There is a scenario in which we can map multiple products with same tracking Id, and hence we need to trigger multiple web service request to the client. Since the tracking Id is unique, we modified its format and introduced a different structure), but the client team doesn't agree with this since it will break length restriction checks at their database. But they are fine to use alphanumerals. One of the solutions we proposed is to use the base 16 encoding (convert that number ("123456") to hex (we will get "1E240")), so the final representation of AB-123456-1 will become AB-1E2401. We would like to check whether this is the right approach and whether there is an alternate solution available for this type of problem.

The hex representation stays invisible to the customers. It is an indirect customer transaction and is not rendered in UI (it is a web service call). Our client uses this field (specifically the numeric field) in their databases as a unique field. As our client saves it as a string in the database, converting to hex is ok for them, but it is more work for them if we are increasing the field length. In our end, we just receive the data, process it and send to the client. Our client uses - as the separator and the 123456 as the unique field in the database. We are fine to have the additional suffix limited to single-digit as long as it is easily identifiable programmatically. There is a meaning behind the suffix, it indicates a separate product which belongs to same tracking Id (Other information like cost of the product will differ in the request we send them).

  • 3
    There is only one solution and it's not technical: get the client team to cooperate. – whatsisname Sep 5 '19 at 16:26
  • 1
    Are you talking of a client team which maintains the UI for the users, and the suggestion is to change the number from decimal to hex at the UI level for the users??? Seriously? Or do you have something different in mind, where the hex representation will stay invisible? – Doc Brown Sep 5 '19 at 16:29
  • 1
    It would heavily improve your post when you edit this information into the question. – Doc Brown Sep 5 '19 at 17:04
  • 1
    Thank you. I have updated the question with this detail – Suraj Muraleedharan Sep 5 '19 at 17:33
  • 3
    It sounds like a bit of a mess to me. If the field itself cannot be lengthened but is capable of accepting any string, then it would appear that the only possible option within that constraint, is to use an encoding larger than base 10. If it will not be rendered to users but only processed computationally, then base 16 is probably not a bad choice - but hex values of 6 characters long would be a bad choice if they are seen and handled by users, in which case a purely alphabetic encoding, or alphabetic prefix and numeric suffix, would probably be better than mixed alphanumeric. – Steve Sep 5 '19 at 18:04
1

When this really solves the problem, why not? Sometimes, a pragmatic solution with some "duct tape" is required to get things done, even if it does not look beautiful from a design perspective.

However, if the other party has already stored these keys using decimal digits in their database, does the new encoding not collide with the existing keys? If you send them a number "123456" as a hex number, and they have already used this decimal as a key in their db, doesn't that lead to conflicts when they now try to store different record with exactly this key again?

In case the other party can handle this, go ahead. If they can change the encoding now, they can probably change it later to something which is even more compact like "base 64", in case "base 16" won't be sufficient any more. And maybe that gives them enough time for making their system a little bit more evolvable, so they can handle trivial field length extensions in a more sane way.

2

This is precisely why storing business defined identifiers as database identifiers becomes problematic. The business changes their rules, and this forces downstream changes in databases.

If you were to rewind time using a UUID (or GUID in .NET terminology) would be an ideal identifier for the same data between two systems, and keep the "business Id" as a separate field. Clearly you cannot do this right now, but you can do this half way.

You can store two Ids for each record. One is the "business friendly" Id, and the other could be your hex-encoded Id you send to the client service. This works OK if your system isn't using the business friendly Id as foreign key values in your own tables.

This way the client service gets what they want, and you can still keep a more technical friendly Id for integrating services.

I would even go a step further and add a UUID Id column internally, and reference that as a foreign key in conjunction with the business friendly Id and the client service friendly Id. This will set you up for a future where the UUID can be passed between services, which is ultimately tied to the business friendly Id. The business friendly Id should be stored in just 1 table with the UUID being the primary key and foreign key values within your system, and being the identifier used by outside systems.

  • Definitely, agree with your point. But the system we work on does not persist this field in the database, we just process it and send to our client (as a web service request). And they have some process at their end and finally save it to their database. – Suraj Muraleedharan Sep 5 '19 at 17:36
  • @SurajMuraleedharan: If you are not saving this Id to your database, then what is the point in trying to work around this issue? Sounds like you should just make the necessary transformation and send the data on its way. Is your system forwarding data to any other client systems, or just this one? – Greg Burghardt Sep 5 '19 at 17:38
  • At our client-side, they are saving this Id to their database. We are trying to make this functionality implemented with no impact (and no change) at the client-side. The client does not agree to receive this field in a different format. – Suraj Muraleedharan Sep 5 '19 at 17:44
  • +1 add more fields, don't try and cram multiple bits of info into a single string – Ewan Sep 6 '19 at 14:01
2

Encode the number as base 64 using a fixed size representation and restrict one of the characters to be non-numeric to prevent collisions with existing identifiers. Base 64 uses 'A'-'Z' to represent 0-25, 'a'-'z' to represent 26-51, '0'-'9' to represent 52-61, '+' and '/' to represent 62 and 63.

For example use 4 characters for the number and 2 for the suffix, with the first character for the suffix in the [A-Za-z] range. That allows 64^4 ~ 16.7 millions of values for the number and 52x64 = 3328 values for the suffix.

123456-1 (why not start the suffix at 0?) becomes the indexes [0][30][9][0]-[0][1] in the base 64 table as 123456=0x64^3+30x64^2+9x64^1+0x64^0, with the last 2 values representing the suffix, so the result is AeJAAB.

If you are really sure that you will never have more than 52 values (or 54 if you use the '+' and '/') for the suffix you can use 5 characters for the number so you have ~ 1071 millions of possible values.

1

Unclear elements

It would be useful to clarify the following points:

  1. Is the - currently stored in the string ?
  2. Is the length control on the client side based on the visible string or on the length of the data sent to the backend ?
  3. Is the unique identifier a concatenation of two values (i.e. could there be AB-123456 and CD-123456 ? Or is 123456 the real identifier and AB is just an additional info ?)
  4. Is the additional suffix -1 limited to a single digit ?
  5. Is the additional suffix an arbitrary extension of the id length (i.e. the separator is just here to facilitate reading) ? Or is there a meaning behind this suffix (e.g. would there a a conceptual relation between AB-123456-1 and AB-123456-2) ?

Suggestions

If answer to (1) is true, then the easiest approach would be to get rid of the separator:

AB-123456
AB1234561  instead of AB-123456-1

If answer to (2) is visible, then you have a reason more to get rid of the separator. But really, the front-end team should be more flexible : in the 80's it was a big deal to make a field longer, but in the XXIst century, really ?

If answer to (2) is backend, then you have a reason to get rid of the separator in the DB, but adatpt the display to insert the - where the user expect them.

In other cases, of course, going hexadecimal is a way to compress the central number. But you gain only one char, since you'll still need 5 hex digits to represent 999999. If you go for a base 32 instead of a base 16, you could spare 2 chars, encoding your number on 4 digits.

Note that you cannot find an encoding scheme that is smaller than 4 printable characters. With a base 95 (using as digit all the printable chars of the ascii character set), you'd still need 4 chars. If non printable chars would be tolerated, you could compress the number on 3 chars (since 20 bits are needed to encode a 6 decimal digit number).

  • 1
    "in the 80's it was a big deal to make a field longer, but in the XXIst century, really ?" - and still COBOL is in the top 50 of the Tiobe index. I guess we would all be astonished if we knew how many businesses still rely on huge IT systems from the last century. – Doc Brown Sep 5 '19 at 19:47
  • True ! But I still have difficulty to imagine COBOL for the front-end ;-) – Christophe Sep 5 '19 at 19:50
  • Front-end? Who knows what's working behind the web service mentioned by the OP? – Doc Brown Sep 5 '19 at 19:53
  • Thanks for the note, I just edited the question. Hope it is more clear now. – Suraj Muraleedharan Sep 7 '19 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.