Let’s turn, now, to whether we should expect to actually see practically PS-misaligned APS systems
deployed in the world.
The previous section doesn’t settle this. In particular: if a technology is difﬁcult to make safe, this
doesn’t mean that lots of people will use it in unsafe ways. Rather, they might adjust their usage
to reﬂect the degree of safety achieved. Thus, if we couldn’t build planes that reliably don’t crash,
we wouldn’t expect to see people dying in plane crashes all the time (especially not after initial
accidents); rather, we’d expect to see people not ﬂying. And such caution becomes more likely as the
stakes of safety failures increase.
Absent counterargument, we might expect something similar with AI. Indeed, some amount of
alignment seems like a signiﬁcant constraint on the usefulness and commercial viability of AI
technology generally. Thus, if problems with proxies, or search, make it difﬁcult to give house-
Is Power-Seeking AI an Existential Risk?