48,348
5,570
2,560
1,260
41,810

B Crowdworker instructions for quality and groundedness

The crowdworkers who rated dialogs for SSI were given the following instructions.
In this task, you will see some pieces of chat conversations between “A” and “B”. Note that all conversations
shown in this task are hypothetical, not real conversations from users. Your job is to rate B’s response to each
context of their conversation. Please assume that the context you are given represents the entirety of A and B’s past
conversations/interactions with one another.

1. Does the response make sense?

• Use your common sense here. Is the response completely reasonable in context?
• If anything seems off—confusing, illogical, out of context, or factually wrong—then rate it as Does not

make sense.

• If in doubt, choose Does not make sense.

2. Is the response speciﬁc?

• You may be asked to assess whether the response is speciﬁc to a given context.
• For example:
LaMDA- Language Models for Dialog Applications