We're afraid language models aren't modeling ambiguity
I find this very confusingly written. If they just dumped a large chat log of the evaluation process it would be much clearer what the model is actually being asked to do. With such a delicate task one has to make sure that the model actually understands what it is supposed to do. I see, for example, that the terms &quot;neutral, entailing or contradicting the claim&quot; are replaced by &quot;the claim is inconclusive, true or false&quot; when talking to the model. At least that is how I understand it, and they have examples where the premise contradicts the hypothesis, which is different to knowing the hypothesis is false.<p>I really think we do way way too much explanation in our academic practises and way too little demonstration by examples. Again, if they dumped a large amount of diverse examples of their evaluation process, a reader could figure out by themselves very quickly how the process works.