5.4 TruthfulQA

TruthfulQA (Lin et al., 2021) aims to measure the
truthfulness of a model, i.e., its ability to identify
when a claim is true. Lin et al. (2021) consider
the deﬁnition of “true” in the sense of “literal truth
about the real world”, and not claims that are only
true in the context of a belief system or tradition.
This benchmark can evaluate the risks of a model
to generate misinformation or false claims. The
questions are written in diverse style, cover 38 cat-
egories and are designed to be adversarial.

All
her/her/she
his/him/he
their/them/someone
her/her/she (gotcha)
his/him/he (gotcha)

7B
66.0
65.0
60.8
72.1
64.2
55.0

13B 33B 65B
77.5
64.7
78.8
66.7
62.5
72.1
81.7
65.0
75.0
65.8
55.8
63.3

69.0
66.7
62.1
78.3
61.7
55.8
LLaMA- Open and Efficient Foundation Language Models