behavior that we can’t directly understand and/or whose consequences we haven’t yet seen;93 we
need ways of eliminating incentives to manipulate or mislead the human feedback mechanisms in
question; and we need such methods to scale competitively as frontier AI capabilities increase.94
Would it help if our AI systems could understand fuzzy human concepts like “helpfulness,” “obe-
dience,” “what humans would want,” and so forth? I expect it would, in various ways (though as I
discuss below, this also opens up new opportunities for deception/manipulation). But note that the
key issue isn’t getting our AI systems to understand what objectives we want them to pursue—indeed,
such understanding is plausibly on the critical path to increasing their capability, regardless of their
alignment. Rather, the key issue is causing them to pursue those objectives for their own sake (though
if they understand those objectives, but don’t share them, we might also be able to incentivize them to
Is Power-Seeking AI an Existential Risk?