Summary: Recent posts like The Neuroscience of Desire and To
what degree do we have goals? have explored the question of
whether humans have desires (or 'goals'). If we don't have
desires, how can we tell an AI what kind of world we 'want'?
Recent work in economics and neuroscience has clarified the
nature of this problem. In 1981, K&T found that human choice was
not always guided by the objective value of possible outcomes,
but by the way those outcomes were 'framed'.1 For example in one
study, K&T told subjects the following story: Imagine that the
U.S. is preparing for the outbreak of an unusual Asian disease,
which is expected to kill 600 people. Two alternative programs
to combat the disease have been proposed. Half the participants
were given the following choice: If program A is adopted, 200
people will be saved. If Program B is adopted, there is a 1/3
probability that 600 people will be saved and a 2/3 probability
that no people will be saved. The second half of participants
were given a different cho