Trapped priors as a basic problem of rationality
Rationality in service to what end, though? You can&#x27;t just build some complicated instruments to record measurements of the ideal polity, the way you can refine measurements of the electron&#x27;s mass. Most political disagreements are about values more than wonkish policymaking.<p><i>But in fact many political zealots never accept reality. It&#x27;s not just that they&#x27;re inherently skeptical of what the other party says. It&#x27;s that even when something is proven beyond a shadow of a doubt, they still won&#x27;t believe it.</i><p>This could describe cognitive error [1] or it could describe someone who&#x27;s using deontological reasoning more than consequentialist reasoning. (And perhaps lacking the vocabulary to say &quot;consequentialist arguments won&#x27;t sway my values.&quot;)<p>It is probably a good strategy to refuse to be talked into positions against your fundamental values by mere evidence. I&#x27;m joking but also serious. I have a vague sense that torturing suspects doesn&#x27;t work to prevent terrorist attacks, but that&#x27;s not the root of my opposition to torture. Maybe clever torturers could make it work more often than not. That wouldn&#x27;t sway me, because torturing suspects is against my values. Documenting how well it can &quot;work&quot; won&#x27;t make me reconsider.<p>[1] Further food for thought: &quot;cognitive error&quot; and &quot;revealed preference&quot; may also be different names for the same thing. Are they experimentally distinguishable?