I'm making a C program that uses GTK3 (best documentation ever) and OpenSSL (worst documentation ever) to simply encrypt/decrypt a file using a password. The encryption and decryption functions are both working, however if I enter the wrong password in the decryption phase, the file becomes corrupt (obviously). So I need a way for the decryptor to detect if it's job was successful. Here are my solutions so far:

  • AFTER encrypting, prepend a check-sum (sha1/sha256) of the original data to the file. So the decryptor can validate the original checksum and decrypted data's checksum.

  • BEFORE encrypting, prepend a constant value (ie "CHECK STRING") to the file, and if the decryptor see's that the constant value was successfully decrypted, it can assume the entire file was too.

Can the unencrypted checksum be cross-analysed with the encrypted data in order to speed a brute force?

Could the encrypted constant value, who's unencrypted value is known by the attacker, reveal to the attacker a limited number of passwords that could encrypt the constant in that manner?

I hope this makes sense, I hate being esoteric.

  • 1
    Can the unencrypted checksum be cross-analysed with the encrypted data in order to speed a brute force? -- No. – Robert Harvey Nov 30 '16 at 3:23
  • 1
    Could the encrypted constant value, who's unencrypted value is known by the attacker, reveal to the attacker a limited number of passwords that could encrypt the constant in that manner? -- No. – Robert Harvey Nov 30 '16 at 3:24
  • Since you're talking about password based encryption, I want to note that you should not use a password directly as key. You need to use an expensive, salted password based key derivation function, such as PBKDF2, scrypt, bcrypt or Argon2 to derive a master key from the password. – CodesInChaos Nov 30 '16 at 16:07
  • I'd consider using LibSodium over OpenSSL. – CodesInChaos Nov 30 '16 at 16:08
  • @CodesInChaos I am using PKCS5_PBKDF2_HMAC. – Dellowar Nov 30 '16 at 16:23

I recommend using authenticated encryption. Its main purpose is that an attacker manipulating the ciphertext can't change the message without getting detected. But it will also reject incorrect keys.

You can either:

  • Use an existing authenticated encryption algorithm, like AES-GCM, AES-CCM or XSalsa20Poly1305.
  • Build one from encryption and a MAC using the encrypt-then-MAC construction.

    A MAC is similar to a hash, but keyed. You can only compute its output if you know the key.

Other constructions, like the ones you mention in the question are not secure when combined with common encryption algorithms like AES in CBC mode. The best known weakness is the padding oracle attack against unauthenticated CBC mode, but similar attacks apply to many such ad-hoc constructions.

If you want you can store a value derived from the master key (e.g. using HKDF) in the header. That can help distinguish between an incorrect key and a corrupted/truncated file.

| improve this answer | |
  • Based on my question, it's apparent that I don't know what a MAC is. (I do now after some research). But for future readers, could you please elaborate what a MAC is and how one is created? Or perhaps explain the process of AES-GCM in detail? Thanks! – Dellowar Nov 30 '16 at 15:51
  • This is extremely important. It's trivial to modify a ciphertext in a way that produces predictable changes in the plaintext. Encryption doesn't provide integrity. You must have some mechanism that assures you the ciphertext hasn't been tampered with before you even try to decrypt it. – Doval Nov 30 '16 at 18:50

I think, your approach of appending or prepending the checksum would be a more solid approach than, using a magic check string.

Reason - You cannot be completely sure that a garbage sequence generated by decryptor when using a wrong password will not contain the magic string.Though the probability might be low. It's possible to run out of luck.

| improve this answer | |
  • 1
    You can't be sure that a decryption with an incorrect key won't produce data that coincidentally hashes to the same value, either. If the check string has a similar length to the hash, the probabilities will be about the same. In reality, neither will happen. – Periata Breatta Nov 30 '16 at 18:58

A simple way to be rather sure encrypt like this.

  • create a hash for the file,
  • append the hash to the file and encrypt this sum.

On decrypt:

  • after decrypt spilt the added hash,
  • re-create the hash for the real file and see if it's the same as the decrypted one.

Since you add a hash you will not add additional information for a brute force attack. A hash though will allow to be be almost (!) 100% sure that the decryption was ok.

| improve this answer | |
  • That helps against accidential corruption, but not against deliberate manipulation of the ciphertext. Depending on the use case it might even turn into a decryption oracle. – CodesInChaos Nov 30 '16 at 9:55
  • @CodesInChaos No. You can not use a hash to get hints. A hash is the result of a one way function. Only if you use magic numbers or check sums you could use it for the decryption oracle. – qwerty_so Nov 30 '16 at 10:26
  • Yeah, this sounds best to me. And sounds just like the OP's "AFTER encrypting" strategy, except you're appending the hash before encrypting, so the hash value is itself encrypted along with the user's cleartext. And despite RobertHarvey's subsequent comments that knowledge of an unencrypted hash can't help attackers, why take any chances? I don't even see why the OP said "AFTER" in the first place. – John Forkosh Nov 30 '16 at 11:09
  • @ThomasKilian Depends on how you do it. But MAC-then-encrypt or even worse, hash-then-encrypt are often broken. For example padding oracles still work, because padding gets checked before the hash. Hash-then-encrypt is even more broken with many encryption algorithm. Even if you're lucky enough to avoid all decryption oracles, it still might not protect you from accepting manipulated files. – CodesInChaos Nov 30 '16 at 12:07
  • @CodesInChaos I have to admit that I'm not deeply in these matters. But AES-GCM does not seem to be implemented in official sources, so you need to implement it according to the RFC/NIST specs. I guess it always depends on the level of security you need. I'd think that this approach is easily feasible and does not offer too much lever for breaking it. – qwerty_so Nov 30 '16 at 12:33

I would say that any input file needs to be verified and cannot just be trusted. For example if you have an input file that is supposed to be a JPEG file, something in your code needs to check this and turn it into an image safely or fail safely, with no attack possible to put some unexpected data into the fail.

If you do that, then there is no need to check whether the file was correctly decrypted - the result of decryption would be a file that is supposed to be a JPEG file in this example, and most likely isn't if the password was wrong. If you properly check the contents of the file, then you will find that it cannot be turned into an image, and that is all you need to know.

| improve this answer | |
  • 1
    Sounds like the answer with the 4 down votes. – qwerty_so Nov 30 '16 at 9:16
  • 1
    A crypto program doesn't and shouldn't know what kind of file it decrypts. The user deleting the ciphertext after successful decryption isn't unlikely. Without proper integrity checks the user might only realize later that some of their files are corrupted (e.g. because they encrypted the important files using a different password). – CodesInChaos Nov 30 '16 at 10:13
  • Also if you process files manipulated by an attacker, this will likely lead to decryption oracles, like the famous padding oracle. – CodesInChaos Nov 30 '16 at 10:15

Rather than trying to check the decryption. Why not pass the file to something that can read it. ie an API or application. If it failed decryption, the software won't be able to read it and should generate an error.

| improve this answer | |
  • 1
    Well the file itself could be ANYTHING. Like a word doc, CSV, a spaghetti recipe, there would be no catch-all. It could even be an already encrypted file. – Dellowar Nov 30 '16 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.