If I want to do file encryption in a desktop piece of software I'm assuming I'll have to hardcode an encryption key in my compiled code. Is that assumption correct and if so what precautions can I take to protect the key?
There's nothing you can do to "protect" the encryption key so long as the key exists somewhere in a user's environment.
Here's some possible attack vectors:
- I could scan the memory of the computer and extract the key once it was loaded into RAM.
- I could look through your program's resources and find anything (such as a string) that resembles your key.
- I could reverse-engineer your program to find out how the key is loaded, then reproduce the same steps to load the key myself.
As long as the encryption key exists in an environment that a potential hacker has control over, said hacker can modify the environment in any way necessary to find your key. The only possible way of preventing said hacker from getting the key would be to isolate the encryption in an environment that the hacker couldn't control. A couple possible solutions:
- A hardware-based device with the key built-in that does the encryption for you. Measures would have to be taken to protect the physical device.
- A server-based solution where the data is uploaded, encrypted, then downloaded.
- An asymmetric encryption scheme, where a "public" key is used to encrypt, and a "private" key is used to decrypt (or the other way around), though this depends on the purpose of "hiding" the keys.