Thoughts on Data Encryption Strategy for Large Number of Records MongoDB and Azure Key Vault

Question

I have previously implement Client Side Data Encryption using Azure Key Vault using the following approach:

Encryption Approach:

1. Every record that needs to be encrypted gets a Content Encryption Key (CEK)
2. The content is encrypted symmetrically using this CEK.
3. The CEK is then encrypted asymmetrically using a Master Key which is stored in Azure Key Vault.
4. The encrypted CEK, the Master Key identifier and the encrypted data are persisted to the Mongo DB.

Decryption Approach: 1. When a User needs decrypted information, the encrypted CEK is first decrypted using the Master Key 2. The decrypted CEK is then used to symmetrically decrypt the encrypted data.

This approach works great if you want to decrypt records one record at a time.

This approach has a performance overhead when you consider decrypting 100s of records at the same time.

For each record, you need to first decrypt the CEK. This is an expensive call over the network using Azure Key Vault Rest API. Imagine someone wanting to export 1000s of decrypted records (as is the case in my scenario now). This would be not just a time consuming operation, but also an expensive one.

Does anyone have any suggestions on how to accomplish this? I have dabbled with the following approaches:

1. Establish a Key Daily (or Monthly) and use it as a CEK for all of the Day/Month's Orders so that the number of CEKs to decrypt reduces when exporting/pulling reports to view en-masse. Use in-memory caching strategies to cache decrypted CEKs and only make a network call when not decrypted CEK not found in cache.

2. Establish a one time CEK for the Client and use it for all records.

3. Don't use a CEK at all and simply encrypt all data using a master key (suicide in my opinion as asymmetric encryption of large data is not recommended at all) and will increase overheads.

Wondering if people have any suggestions!

Answer 1

Option 1 you suggested seems like a reasonable approach. When we had a very similar problem (different technologies, but same encrypted key pattern) we gave data that was always queried together the same CEK (in our case data for one user). We used a microcache to hold the CEKs. Basically we cached each CEK with an expiry of only 15 seconds, since most of the time we would use the key a bunch in a short time period, then not at all.

I'd also recommend doing as much in memory caching of the decrypted data as possible. Generally speaking, data in memory is reliably inaccessible to other applications, as long as you aren't using some sort of shared caching tool. Depending on your use case, you might be able to hold a pretty substantial part of your working data set in memory permanently.