Whats safest way to tell if decryption was successful?

Question

I'm making a C program that uses GTK3 (best documentation ever) and OpenSSL (worst documentation ever) to simply encrypt/decrypt a file using a password. The encryption and decryption functions are both working, however if I enter the wrong password in the decryption phase, the file becomes corrupt (obviously). So I need a way for the decryptor to detect if it's job was successful. Here are my solutions so far:

• AFTER encrypting, prepend a check-sum (sha1/sha256) of the original data to the file. So the decryptor can validate the original checksum and decrypted data's checksum.

• BEFORE encrypting, prepend a constant value (ie "CHECK STRING") to the file, and if the decryptor see's that the constant value was successfully decrypted, it can assume the entire file was too.

Can the unencrypted checksum be cross-analysed with the encrypted data in order to speed a brute force?

Could the encrypted constant value, who's unencrypted value is known by the attacker, reveal to the attacker a limited number of passwords that could encrypt the constant in that manner?

I hope this makes sense, I hate being esoteric.

Answer 1

I recommend using authenticated encryption. Its main purpose is that an attacker manipulating the ciphertext can't change the message without getting detected. But it will also reject incorrect keys.

You can either:

• Use an existing authenticated encryption algorithm, like AES-GCM, AES-CCM or XSalsa20Poly1305.

• Build one from encryption and a MAC using the encrypt-then-MAC construction.

  A MAC is similar to a hash, but keyed. You can only compute its output if you know the key.

Other constructions, like the ones you mention in the question are not secure when combined with common encryption algorithms like AES in CBC mode. The best known weakness is the padding oracle attack against unauthenticated CBC mode, but similar attacks apply to many such ad-hoc constructions.

If you want you can store a value derived from the master key (e.g. using HKDF) in the header. That can help distinguish between an incorrect key and a corrupted/truncated file.

Answer 2

I think, your approach of appending or prepending the checksum would be a more solid approach than, using a magic check string.

Reason - You cannot be completely sure that a garbage sequence generated by decryptor when using a wrong password will not contain the magic string.Though the probability might be low. It's possible to run out of luck.

Answer 3

I would say that any input file needs to be verified and cannot just be trusted. For example if you have an input file that is supposed to be a JPEG file, something in your code needs to check this and turn it into an image safely or fail safely, with no attack possible to put some unexpected data into the fail.

If you do that, then there is no need to check whether the file was correctly decrypted - the result of decryption would be a file that is supposed to be a JPEG file in this example, and most likely isn't if the password was wrong. If you properly check the contents of the file, then you will find that it cannot be turned into an image, and that is all you need to know.

Answer 4

Rather than trying to check the decryption. Why not pass the file to something that can read it. ie an API or application. If it failed decryption, the software won't be able to read it and should generate an error.