How should I manage secrets for applications deployed on servers I do not control?

Question

Problem description

I have an IoT app that provides a server backend and browser UI client for interfacing with proprietary control systems. Some deployments of this application will require that the server is not publicly accessible on the internet; instead, the hardware will often be part of a network managed by the end-user. To complicate matters further, some end-users will not have any external internet access provided to the server, whilst others will allow some token-based access on an ad-hoc basis for debugging and maintenance.

The app requires interfacing with public 3rd party APIs, e.g., sending emails (this functionality will not work for the users who will not provide an internet connection to the server). I have already containerised the application with Docker (using docker-compose, and the API keys are not committed to any version control system.

I understand that securing these keys is difficult, if not impossible, as physical ownership of the server is with the end-user. In light of this, I’d like to know what options and strategies are at my disposal to mitigate the inherently risky deployment solution. I have presented some of my thoughts below.

Option 1: Separate API keys per end-user

One option is to generate a fresh set of secrets for every deployment. E.g. no two end-users use duplicate email API keys. This would allow more fine-grained control of an end-user’s interaction with the 3rd party APIs and respect API provider rate limiting.

The obvious disadvantage to this is it does not scale well. Haven’t to maintain m keys for every n end-users seems though it could grow out of control quickly.

Option 2: Commissioning the physical server in-house

We can deploy the application ourselves on the end-user’s hardware before physically shipping it to the deployment location. In some cases, this is unavoidable as we cannot always remotely access the server to deploy the app.

I’d like to know of any other options this opens up for addressing my issue.

Option 3: All 3rd party interactions are routed through a public middleman service

A publicly hosted service could be used to handle all of the external API connections. For example, the deployed application doesn’t ever interact with the emailing API directly. Instead, it fires a message to this middleman service to send the email, and receives a message if and when the email is sent. Does anything like this already exist, or is this a case of rolling your own? I would prefer avoiding having to add extra infrastructure if possible.

Answer 1

Taking these slightly out of order:

• Option 2: build the server in house. This doesn't fundamentally change things - the server with the keys on it will still be in the physical control of your customers, and with sufficient effort they will be able to extract the secrets.

• Option 1: automation solves many things. In whatever pipeline you have for creating the machine (image) for a new customer, add deploy-secrets.sh > /etc/secrets.

• Option 3: you're still going to need a secret for your middleman service per customer; this does reduce your scaling problem from m APIs × n customers to just having 1 secret × n customers. If you asked me to stand up a solution like this, I'd have a serverless function running in one of the major public clouds; whether that makes sense for you depends on your business.

Only you can evaluate how much effort and how much benefit each of these options brings; it may be that Option 2, by raising the bar to extract the secrets, is good enough for your business's needs. If it's not, you need to decide between Option 1 and Option 3.

Answer 2

I'd probably go for option 1, even though it does have its shortcomings.

The main reasoning is that an API key (like other credentials) gives access to a service but also expresses who is responsible for the use or misuse of the service.

If your application runs on a server you control, it's essentially you who is responsible for proper use of external services, so a single API key bound to the application is fine.

However, if the server is accessible by third party it depends on trust whether you want to assume responsibility for their actions or not. Trust does not only cover the assumption that customers won't do anything malicious, but also that they are able to keep entrusted credentials safe. If the credentials are their own, they explicitly assume responsibility for their own actions. Of course, they then need to trust your software to not perform any unwanted activity in their name.

Option 3 enables you to simplify the credential management per customer (only give them one credential which allows them to access your middleman) and also makes it possible to cut off access to all third party services when the contract ends. However, it does force you to set up and maintain additional infrastructure and creates a possible additional point of failure.

At the end of the day, you need to decide which cost/risk combination is ok for you.

Answer 3

A problem you've not mentioned is revocation and reissue. What happens when you need to change the keys?

Option 3 is most secure: can't leak a secret they don't have, you control all the keys, and can change them at will.

Option 2: doesn't actually make any difference as it stands. What might make a difference is using some kind of hardware security module or dongle to store the keys, but for API requests I don't think there's a standardized solution.

I suggest that you take a leaf out of the popular and hugely successful Let's Encrypt service. It has to be adapted because this isn't PKI.

• Create keys for everyone automatically. Store them in your database, carefully.

• On bootup, each edge device presents its ID and receives keys. There is an expiry time associated with these after which they will be re-requested.

• Abuse can be traced to a key and therefore customer

• If you need to revoke a key, you can just do so. The affected customer can get a new one or not depending on whether you're also cutting them off.