(May 2022 author’s note: since making this report public in April 2021, my probability estimates –
discussed in this section – have shifted. My current overall probability of existential catastrophe from
power-seeking AI by 2070 is >10%.)
To sum up, and with the preceding discussion in mind, let’s return to the full argument I opened with.
To illustrate my current (unstable, subjective) epistemic relationship to the premises of this argument,
I’ll add some provisional credences to each, along with a few words of explanation.170
To be clear: I don’t think I’ve formulated these premises with adequate precision to really “forecast”
in the sense relevant to e.g. prediction markets; and in general, the numbers here (and the exercise
more broadly) should be held very lightly. I’m offering these quantitative probabilities mostly
because I think that doing so is preferable, relative to leaving things in purely qualitative terms
Is Power-Seeking AI an Existential Risk?