1

I have an application that involves processing packets of IDs from a variety of sources. Some of these sources contain the information I want, some of them constitute, effectively, noise. Currently, whenever my application receives a data packet, it checks the database to verify that the received IDs match internal data before doing more work. I would like to eliminate this step and/or minimize the processing due to noise.

One idea I had was to make some portion of my IDs non random. E.g. instead of using a completely random UUID, I might replace the last four characters with a fixed string - thus my application can perform a simple check that will be able to easily filter out the noise 99.9% of the time.

However, this seems... dirty...

Another idea would be to create hashes from some bit of random string and some other constant. Hence when un-hashed, I could locally detect if the constant matched. However, this just seems like adding a layer of complexity on top of the dirty solution.

How should I best deal with this situation? Am I right in my instinct that my proposed solution is a potentially bad idea?


Each time my application (a mobile app) receives a packet (bluetooth low energy beacon data), it sends a request to a server (AWS lambda function), which then queries the database (DynamoDB) to find the ID and responds with the result of the query. Servers, database throughput, and API calls cost money. Hence, reducing the number of times I have to perform this operation reduces my costs. Furthermore, I feel that minimizing the total amount of bandwidth I'm requiring from my users mobile phones is just a nice thing to do.

  • Can you clarify the phrase "verify that the received IDs match internal data"? Also, what does "minimize processing" mean? Are you having performance problems that are directly impacting the application's ability to perform its job? More specifically, what use case/user story is achieved or improved by implementing your proposed modification, and how? – Robert Harvey Jan 23 '17 at 23:28
  • 2
    Well, I saw your edit. I'd say that, if the technique that you propose meets your stated goals without compromising security or causing other problems, then it's the "right" thing to do. – Robert Harvey Jan 24 '17 at 1:53
  • can you tell us a little more about how the Ids are assigned, where, and what the threat profile for hacking is? – Paul Jan 24 '17 at 2:06
  • How do you determine if a packet is noise? If you can do it locally, it seems a good idea to filter out noise without touching a DB. – 9000 Jan 24 '17 at 16:41
  • Couple of questions: Hashes are one-way so what do you mean by "unhash"? Is there some reason that anything in this packet must be considered an id? Can you add non-id data to the packet? – JimmyJames Jan 24 '17 at 17:35
2

This is a very broad question, and you haven't stated many of the things I would need to know to tell you what is better and what is not. But here are things to consider:

  1. Is it a requirement that all identifiers be globally unique? Who owns and enforces the namespace? By embedding the source of the ID, you can make it easier to ensure uniqueness, because each source has its own mini-namespace.

  2. Does the size of the identifier matter much? A longer identifier of course takes up more space, and can be more difficult to search for in certain data structures, e.g. a DB index, which will be able to fit fewer identifiers per index page.

  3. Are the identifiers tamper proof? By changing the data portion of the ID, could a hacker make a packet do something that it isn't normally supposed to do?

  4. Are the identifiers guessable? By picking other IDs, could a hacker do something nefarious? By introducing a data portion to the ID, you may be reducing its entropy.

  5. Do you have any requirements for forward- or backward-compatiblity?

  6. How are the IDs transmitted? Is there a limited character set, or will anything need encoding or escaping?

  7. Is the current ID fully numeric, and are you proposing adding non-numeric characters to it? Non-numeric IDs are more vulnerable to injection attacks.

  8. Are there any requirements for sortability?

  • -1 - all imporant questions to ask but this does not answer the question – marstato Jan 24 '17 at 12:23
  • you haven't stated many of the things I would need to know to tell you what is better and what is not – John Wu Jan 24 '17 at 13:17
  • @marstato: These questions help the OP answer the original question. Without good answers to this list, a good answer to the original question is hardly possible. – 9000 Jan 24 '17 at 16:37
1

I am answering you from a database developer perspective. Generally speaking the best practice is to use a sequential integer for an ID. Primary reason is performance, namely the way the database chunks the data when you insert (page and cluster or something like that). A DBA can tell you more. Also, from a business perspective, encoding business data in an ID is almost always a bad idea. You run into problems when sorting and joining, etc. Just make a new property on your business object. That is what they are there for. ID's are there to identify.

  • 1
    While I do agree that a Surrogate Key is almost always the best way to go (this is a controversial opinion, BTW), I don't agree that a sequential integer is ideal. In many applications, GUIDs have more favorable characteristics. – Robert Harvey Jan 23 '17 at 23:50
  • @RobertHarvey, I didn't say a sequential integer is ideal (although for many business apps it actually is). I think the OP's question was about embedding business data or metadata in the ID - and that is what I am saying is a bad idea. – Sam Jan 24 '17 at 1:49
  • I think the issue is terminology. The ID he's talking about is not necessarily the primary key of the database records on the server. For scalability reasons, I strongly recommend against both "business keys" and autoincrement integers for DB PK's, but for the purpose he's talking about it's a valid approach. In the actual DB schema the field he's calling the ID would just be some other field (or set of fields) than the primary key. – Paul Jan 24 '17 at 2:03
0

If you want to add something to the IDs that would act as a filter from bad data it probably would be better to include eg. last 4 chars of sha1 or md5 of the ID+salt itself so it can't be easily fabricated.

But I think, however that sending this "checksum" data as another parameter would be better. So you could send ID + checksum, and discard the checksum in the process, or at least save it outside of the ID so you won't add unnecessary data to a key which is indexed, as all data you put there is costly from DB perspective. Best would be if you provide some data sample.

The other part of your question is interesting... because you say that you're going to redesign your application partly because of pricing model of AWS, while for $90/month you can get a server with 32GB RAM, 8 cores and half TB of SSD storage. It's nice to pay for peace of mind if you have money and want to work on something that "just works"... but if you're afraid about the costs, then I think it'd be much better from your perspective to invest your time in moving off of AWS, because the price to performance ratio is so laughable there, that the costs will KILL you in the long run. For me these shared cloud enviroments are always limiting factor, slow, expensive, with crappy, old software installed and bandwidtch priced like crazy.

I mean if you can have millions of requests per second on a server that costs nothing, thinking about every request on AWS is a waste of life, which you can waste on eg. designing new features for your app ... and beside it's pointless, it won't become cheap, no matter how much you castrate the code, compress the communication, etc. it'd be still expensive as HELL for anything but an app that has no traffic ;)

  • What purpose would sending the checksum serve? – Robert Harvey Jan 24 '17 at 1:50
  • If you already want to add some static data to automatically remove noise, why not add 2 lines of code and also prevent fabricated/forged requests ... Not sure if you need it, maybe not... but it won't hurt ;) – Slawek Jan 24 '17 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.