70% on sensibleness, which even surpasses some large dialog models [17].
The second score, speciﬁcity, is used to measure whether a response is speciﬁc to a given context. For example, if a user
says “I love Eurovision” and the model responds “Me too,” then it would score 0 on speciﬁcity, since this response could
be used in many different contexts. If it answers “Me too. I love Eurovision songs,” then it would score 1. Adiwardana
et al. [17] report that Meena narrows the gap to average human performance in the SSA metric.
As the model’s performance increases, however, we ﬁnd that sensibleness and speciﬁcity are not sufﬁcient to measure
the quality of a dialog model. For example, a response to “How do I throw a ball?” could be “You can throw a ball by
ﬁrst picking it up and then throwing it”, which makes sense and is speciﬁc to the question. An alternative deeper and
more satisfying answer could be “One way to toss a ball is to hold it ﬁrmly in both hands and then swing your arm
LaMDA- Language Models for Dialog Applications