5.3.3 for more.

27

A key question for any such strategy, then, is whether it can translate, given success at some level of
capability, into a different strategy that scales better. For example, we might try to achieve practical
PS-alignment with some fairly advanced systems (including, perhaps, quite specialized ones—or,
indeed, non-APS ones), and then use them to create new and superior PS-alignment strategies
(indeed, as AI development itself becomes increasingly automated, automating alignment research
will plausibly be necessary regardless).
But note that plans of the form “create some practically PS-aligned systems, and ask them what the
plan should be” might just not work. For example, the new systems might not have adequate plans
either. One might therefore need to create even more capable systems, which also might not have
adequate plans, and so forth, until one pushes up against (or perhaps, past) the limits of one’s capacity
to ensure practical PS-alignment.
Is Power-Seeking AI an Existential Risk?