Such corrective measures—in conjunction with ongoing work improving our ability to ensure the
practical alignment of the systems we build/deploy—could well be enough to avert catastrophe. But
there are also a number of salient ways they could fail.
One key failure mode arises in scenarios where frontier AI capabilities escalate very rapidly—for
example, because the process of developing and improving frontier AI systems is itself increasingly
automated. The plausibility of scenarios of this kind is a very open question, as are the precise
timescales. But other things equal, more rapid capability escalation provides less time for the world
to get experience with practical alignment failures, and to implement corrective measures; it creates
a larger amount of general civilizational upheaval and disruption; and it may give AI-empowered
actors (including PS-misaligned AI systems themselves) with a smaller calendar time “lead” a larger
absolute advantage over their competitors.
Is Power-Seeking AI an Existential Risk?