Challenge–response authentication
In computer security, challenge–response authentication is a family of protocols in which one party presents a question ("challenge") and another party must provide a valid answer ("response") to be authenticated.

The simplest example of a challenge–response protocol is password authentication, where the challenge is asking for the password and the valid response is the correct password.

Clearly an adversary who can eavesdrop on a password authentication can then authenticate itself in the same way. One solution is to issue multiple passwords, each of them marked with an identifier.  The verifier can ask for any of the passwords, and the prover must have that correct password for that identifier. Assuming that the passwords are chosen independently, an adversary who intercepts one challenge–response message pair has no clues to help with a different challenge at a different time.

For example, when other communications security methods are unavailable, the U.S. military uses the AKAC-1553 TRIAD numeral cipher to authenticate and encrypt some communications.  TRIAD includes a list of three-letter challenge codes, which the verifier is supposed to choose randomly from, and random three-letter responses to them. For added security, each set of codes is only valid for a particular time period which is ordinarily 24 hours.

A more interesting challenge–response technique works as follows. Say, Bob is controlling access to some resource. Alice comes along seeking entry. Bob issues a challenge, perhaps "52w72y". Alice must respond with the one string of characters which "fits" the challenge Bob issued. The "fit" is determined by an algorithm "known" to Bob and Alice. (The correct response might be as simple as "63x83z", with the algorithm changing each character of the challenge using a Caesar cipher). In the real world, the algorithm would be much more complex.) Bob issues a different challenge each time, and thus knowing a previous correct response (even if it is not "hidden" by the means of communication used between Alice and Bob) is of no use.

Other non-cryptographic protocols
Challenge–response protocols are also used to assert things other than knowledge of a secret value. CAPTCHAs, for example, are a sort of variant on the Turing test, meant to determine whether a viewer of a Web application is a real person.  The challenge sent to the viewer is a distorted image of some text, and the viewer responds by typing in that text.  The distortion is designed to make automated optical character recognition (OCR) difficult and preventing a computer program from passing as a human.

Cryptographic techniques
Non-cryptographic authentication were generally adequate in the days before the Internet, when the user could be sure that the system asking for the password was really the system they were trying to access, and that nobody was likely to be eavesdropping on the communication channel to observe the password being entered.  To address the insecure channel problem, a more sophisticated approach is necessary. Many cryptographic solutions involve two-way authentication, where both the user and the system must each convince the other that they know the shared secret (the password), without this secret ever being transmitted in the clear over the communication channel, where eavesdroppers might be lurking.

One way this is done involves using the password as the encryption key to transmit some randomly generated information as the challenge, whereupon the other end must return as its response a similarly encrypted value which is some predetermined function of the originally offered information, thus proving that it was able to decrypt the challenge. For instance, in Kerberos, the challenge is an encrypted integer N, while the response is the encrypted integer N + 1, proving that the other end was able to decrypt the integer N. A hash function can also be applied to a password and a random challenge value to create a response value. Another variation uses a probabilistic model to provide randomized challenges conditioned on model input. 

Such encrypted or hashed exchanges do not directly reveal the password to an eavesdropper. However, they may supply enough information to allow an eavesdropper to deduce what the password is, using a dictionary attack or brute-force attack. The use of information which is randomly generated on each exchange (and where the response is different from the challenge) guards against the possibility of a replay attack, where a malicious intermediary simply records the exchanged data and retransmits it at a later time to fool one end into thinking it has authenticated a new connection attempt from the other.

Authentication protocols usually employ a cryptographic nonce as the challenge to ensure that every challengeresponse sequence is unique. This protects against Eavesdropping with a subsequent replay attack. If it is impractical to implement a true nonce, a strong cryptographically secure pseudorandom number generator and cryptographic hash function can generate challenges that are highly unlikely to occur more than once. It is sometimes important not to use time-based nonces, as these can weaken servers in different time zones and servers with inaccurate clocks. It can also be important to use time-based nonces and synchronized clocks if the application is vulnerable to a delayed message attack. This attack occurs where an attacker copies a transmission whilst blocking it from reaching the destination, allowing them to replay the captured transmission after a delay of their choosing. This is easily accomplished on wireless channels. The time-based nonce can be used to limit the attacker to resending the message but restricted by an expiry time of perhaps less than one second, likely having no effect upon the application and so mitigating the attack.

Mutual authentication is performed using a challengeresponse handshake in both directions; the server ensures that the client knows the secret, and the client also ensures that the server knows the secret, which protects against a rogue server impersonating the real server.

Challenge–response authentication can help solve the problem of exchanging session keys for encryption. Using a key derivation function, the challenge value and the secret may be combined to generate an unpredictable encryption key for the session. This is particularly effective against a man-in-the-middle attack, because the attacker will not be able to derive the session key from the challenge without knowing the secret, and therefore will not be able to decrypt the data stream.

Simple example mutual authentication sequence
Server sends a unique challenge value sc to the client
Client sends a unique challenge value cc to the server
Server computes sr = hash(cc + secret) and sends to the client
Client computes cr = hash(sc + secret) and sends to the server
Server calculates the expected value of cr and ensures the client responded correctly
Client calculates the expected value of sr and ensures the server responded correctly
where
sc is the server-generated challenge
cc is the client-generated challenge
cr is the client response
sr is the server response

Password storage
To avoid storage of passwords, some operating systems (e.g. Unix-type) store a hash of the password rather than storing the password itself. During authentication, the system need only verify that the hash of the password entered matches the hash stored in the password database. This makes it more difficult for an intruder to get the passwords, since the password itself is not stored, and it is very difficult to determine a password that matches a given hash. However, this presents a problem for many (but not all) challengeresponse algorithms, which require both the client and the server to have a shared secret. Since the password itself is not stored, a challengeresponse algorithm will usually have to use the hash of the password as the secret instead of the password itself. In this case, an intruder can use the actual hash, rather than the password, which makes the stored hashes just as sensitive as the actual passwords. SCRAM is a challengeresponse algorithm that avoids this problem.

Examples
Examples of more sophisticated challenge-response algorithms are:
 Zero-knowledge password proof and key agreement systems (such as Secure Remote Password (SRP))
 Challenge-Handshake Authentication Protocol (CHAP) ()
 CRAM-MD5, OCRA: OATH Challenge-Response Algorithm ()
 Salted Challenge Response Authentication Mechanism (SCRAM) ()
 ssh's challenge-response system based on RSA .

Some people consider a CAPTCHA a kind of challenge-response authentication that blocks spambots.

See also

Challenge-handshake authentication protocol
Challenge–response spam filtering
CRAM-MD5
Cryptographic hash function
Cryptographic nonce
Kerberos
Otway–Rees protocol
Needham–Schroeder protocol
Wide Mouth Frog protocol
Password-authenticated key agreement
Salted Challenge Response Authentication Mechanism
SQRL
Distance-bounding protocol
Reflection attack
Replay attack
Man-in-the-middle attack
WebAuthn

References

Authentication methods
Computer access control