Rather, A has to have something like a hypothesis about what H wants (e.g., it asks itself: “would H
want me to maximize apples, or oranges?”), and to act on that hypothesis.64 Intent alignment seems a
promising route to full alignment; but I won’t focus on it exclusively.
I’ll say that a system is “practically aligned” if it does not engage in misaligned behavior on any
of the inputs it will in fact receive.65 Full alignment implies a highly reliable form of practical
alignment—e.g., one that does not depend at all on predicting or controlling the inputs a system
receives. From a safety perspective, this seems a very desirable property, especially given the unique
challenges and stakes that APS systems present (see section 4.4 for discussion). Indeed, if you
are building AI systems that would e.g. harm or seize power over you, due to problems with their
Is Power-Seeking AI an Existential Risk?