We're afraid language models aren't modeling ambiguity
It&#x27;s such a pity that they chose to use PolitFact as labelling &quot;experts&quot; in one of the underlying datasets. Given how extreme the political bias in academia is they should really know better than to try and make claims about politics in what would otherwise be reasonable ML research. The assumption that fact checkers are reliable is going to be an automatic strike against the paper for a large number of readers because it&#x27;s so self-evidently wrong, and calls their judgement into question on everything else.<p>Even though they only give three examples in the paper itself the problems are already obvious. If we look at table 6 then the latter two statements are genuinely ambiguous, but they also provide this as an example of an ambiguous statement: &quot;When President Obama was elected, the market crashed&quot;. This is called ambiguous because &quot;the claim implies a causal relationship&quot;.<p>That statement isn&#x27;t ambiguous. It has exactly one possible parse and meaning, asserting temporal alignment. &quot;When A happened, B happened&quot; doesn&#x27;t automatically imply causality in English, and when people assume it does that&#x27;s the famous &quot;correlation is not causation&quot; fallacy. Yet they classify it as ambiguous because an LLM rewrote it in one case to contain an implied assertion of causality that doesn&#x27;t actually appear in the original text. In effect the political bias of the LLM or possibly the exact data fed in has been used to condemn the original speaker - exactly the sort of thing that makes self-proclaimed &quot;professional fact checkers&quot; such a joke.<p>This sort of problem is inevitable if you use politics as a verification dataset because the internet is full of people reading implied-but-unstated things into factual statements and then &quot;fact checkers&quot; proceed to debunk what they think people <i>would have</i> said, instead of what they actually said.<p>Although politics will have this problem frequently, a few of the other examples in their dev set show similar issues. For example, &quot;Ralph knows that someone here is a spy&quot; is considered ambiguous because you could potentially infer that he either does or doesn&#x27;t know who it is, but you could also argue that the statement is unambiguous and it&#x27;s only your attempt to read more into the statement than actually exists that makes it appear ambiguous. If the original statement was surrounded with more context then that would let you pick between the two possible outcomes but the statement <i>by itself</i> is not ambiguous - it just doesn&#x27;t give as much information as the labeller might have wanted.