Deniable encryption
In cryptography and steganography, plausibly deniable encryption describes encryption techniques where the existence of an encrypted file or message is deniable in the sense that an adversary cannot prove that the plaintext data exists.

The users may convincingly deny that a given piece of data is encrypted, or that they are able to decrypt a given piece of encrypted data, or that some specific encrypted data exists. Such denials may or may not be genuine. For example, it may be impossible to prove that the data is encrypted without the cooperation of the users. If the data is encrypted, the users genuinely may not be able to decrypt it. Deniable encryption serves to undermine an attacker's confidence either that data is encrypted, or that the person in possession of it can decrypt it and provide the associated plaintext.

Function
Deniable encryption makes it impossible to prove the existence of the plaintext message without the proper decryption key. This may be done by allowing an encrypted message to be decrypted to different sensible plaintexts, depending on the key used. This allows the sender to have plausible deniability if compelled to give up their encryption key.
The notion of "deniable encryption" was used by Julian Assange and Ralf Weinmann in the Rubberhose filesystem and explored in detail in a paper by Ran Canetti, Cynthia Dwork, Moni Naor, and Rafail Ostrovsky in 1996.

Scenario
Deniable encryption allows the sender of an encrypted message to deny sending that message. This requires a trusted third party. A possible scenario works like this:

Bob suspects his wife Alice is engaged in adultery. That being the case, Alice wants to communicate with her secret lover Carl. She creates two keys, one intended to be kept secret, the other intended to be sacrificed. She passes the secret key (or both) to Carl.
Alice constructs an innocuous message M1 for Carl (intended to be revealed to Bob in case of discovery) and an incriminating love letter M2 to Carl. She constructs a cipher-text C out of both messages, M1 and M2, and emails it to Carl.
Carl uses his key to decrypt M2 (and possibly M1, in order to read the fake message, too).
Bob finds out about the email to Carl, becomes suspicious and forces Alice to decrypt the message.
Alice uses the sacrificial key and reveals the innocuous message M1 to Bob. Since it is impossible for Bob to know for sure that there might be other messages contained in C, he might assume that there are no other messages (alternatively, Bob may not be familiar with the concept of plausible encryption in the first place, and thus may not be aware it is even possible for C to contain more than one message).

Another possible scenario involves Alice sending the same ciphertext (some secret instructions) to Bob and Carl, to whom she has handed different keys. Bob and Carl are to receive different instructions and must not be able to read each other's instructions. Bob will receive the message first and then forward it to Carl.

Alice constructs the ciphertext out of both messages, M1 and M2, and emails it to Bob.
Bob uses his key to decrypt M1 and isn't able to read M2.
Bob forwards the ciphertext to Carl.
Carl uses his key to decrypt M2 and isn't able to read M1.

Forms of deniable encryption
Normally, ciphertexts decrypt to a single plaintext that is intended to be kept secret. However, one form of deniable encryption allows its users to decrypt the ciphertext to produce a different (innocuous but plausible) plaintext and plausibly claim that it is what they encrypted. The holder of the ciphertext will not be able to differentiate between the true plaintext, and the bogus-claim plaintext. In general, one ciphertext cannot be decrypted to all possible plaintexts unless the key is as large as the plaintext, so it is not practical in most cases for a ciphertext to reveal no information whatsoever about its plaintext. However, some schemes allow decryption to decoy plaintexts that are close to the original in some metric (such as edit distance).

Modern deniable encryption techniques exploit the fact that without the key, it is infeasible to distinguish between ciphertext from block ciphers and data generated by a cryptographically secure pseudorandom number generator (the cipher's pseudorandom permutation properties).

This is used in combination with some decoy data that the user would plausibly want to keep confidential that will be revealed to the attacker, claiming that this is all there is. This is a form of steganography.

If the user does not supply the correct key for the truly secret data, decrypting it will result in apparently random data, indistinguishable from not having stored any particular data there.

One example of deniable encryption is a cryptographic filesystem that employs a concept of abstract "layers", where each layer can be decrypted with a different encryption key. Additionally, special "chaff layers" are filled with random data in order to have plausible deniability of the existence of real layers and their encryption keys. The user can store decoy files on one or more layers while denying the existence of others, claiming that the rest of space is taken up by chaff layers. Physically, these types of filesystems are typically stored in a single directory consisting of equal-length files with filenames that are either randomized (in case they belong to chaff layers), or cryptographic hashes of strings identifying the blocks. The timestamps of these files are always randomized. Examples of this approach include Rubberhose filesystem and PhoneBookFS. 

Another approach used by some conventional disk encryption software suites is creating a second encrypted volume within a container volume. The container volume is first formatted by filling it with encrypted random data, and then initializing a filesystem on it. The user then fills some of the filesystem with legitimate, but plausible-looking decoy files that the user would seem to have an incentive to hide. Next, a new encrypted volume (the hidden volume) is allocated within the free space of the container filesystem which will be used for data the user actually wants to hide. Since an adversary cannot differentiate between encrypted data and the random data used to initialize the outer volume, this inner volume is now undetectable.
LibreCrypt and BestCrypt can have many hidden volumes in a container; TrueCrypt is limited to one hidden volume.

Detection
The existence of hidden encrypted data may be revealed by flaws in the implementation. It may also be revealed by a so-called 'watermarking attack' if an inappropriate cipher mode is used.
The existence of the data may be revealed by it 'leaking' into non-encrypted disk space  where it can be detected by forensic tools.

Doubts have been raised about the level of plausible deniability in 'hidden volumes' – the contents of the "outer" container filesystem have to be 'frozen' in its initial state to prevent the user from corrupting the hidden volume (this can be detected from the access and modification timestamps), which could raise suspicion. This problem can be eliminated by instructing the system not to protect the hidden volume, although this could result in lost data.

Drawbacks
Deniable encryption has been criticized because it does not directly defend users from revealing keys under coercion or torture. Possession of deniable encryption tools could lead attackers to continue torturing a user even after the user has revealed all their keys, because the attackers could not know whether the user had revealed their last key or not. However, knowledge of this fact can disincentivize users from revealing any keys to begin with, since they will never be able to prove to the attacker that they have revealed their last key.

Deniable authentication
Some in-transit encrypted messaging suites, such as Off-the-Record Messaging, offer deniable authentication which gives the participants plausible deniability of their conversations. While deniable authentication is not technically "deniable encryption" in that the encryption of the messages is not denied, its deniability refers to the inability of an adversary to prove that the participants had a conversation or said anything in particular.

This is achieved by the fact that all information necessary to forge messages is appended to the encrypted messages – if an adversary is able to create digitally authentic messages in a conversation (see hash-based message authentication code (HMAC)), they are also able to forge messages in the conversation. This is used in conjunction with perfect forward secrecy to assure that the compromise of encryption keys of individual messages does not compromise additional conversations or messages.

Software
OpenPuff, freeware semi-open-source steganography for MS Windows.
LibreCrypt, opensource transparent disk encryption for MS Windows and PocketPC PDAs that provides both deniable encryption and plausible deniability. Offers an extensive range of encryption options, and doesn't need to be installed before use as long as the user has administrator rights.
Off-the-Record Messaging, a cryptographic technique providing true deniability for instant messaging.
Rubberhose, defunct project (last release in 2000, not compatible with modern Linux distributions)
StegFS, the current successor to the ideas embodied by the Rubberhose and PhoneBookFS filesystems.
VeraCrypt (a successor to a discontinued TrueCrypt), an on-the-fly disk encryption software for Windows, Mac and Linux providing limited deniable encryption and to some extent (due to limitations on the number of hidden volumes which can be created)plausible deniability, without needing to be installed before use as long as the user has full administrator rights.
Vanish, a research prototype implementation of self-destructing data storage.

See also

References

Further reading

Cryptography