&gt; A paperclip making machine could be made without any intelligence involved<p>And very much <i>should</i> be, yes.  It&#x27;s extraordinarily dangerous to build <i>any</i> strong AI (or optimization process) with a value function less general than one that will best satisfy the value functions of sapient beings.  (Side note: that doesn&#x27;t necessarily mean we need to build the AI to itself be sapient; I&#x27;ve seen the term &quot;optimization process&quot; used as a substitute for AI, to imply that it need not have any agency of its own, only a goal to satisfy people&#x27;s goals.)  Any strong AI that <i>doesn&#x27;t</i> specifically value &quot;people&quot; will hurt people as a side effect of maximizing whatever it <i>does</i> value.<p>&gt;&gt; &quot;There is no plausible scenario in which we build a strong AI... and it doesn&#x27;t take over the world&quot;<p>&gt; First off you are using polemic here with the &quot;plausible&quot; card<p>I very specifically mentioned &quot;strong AI&quot;: AI capable of self-improvement to better satisfy some value function.  We could quibble over the word &quot;capable&quot;, but I&#x27;d argue that a computer with the capability of self-improvement that specifically chooses not to do so seems rather pointless; why build the capability if not to use it?<p>&gt; There are in fact no proofs that a strong AI will take over<p>I used &quot;take over&quot; very loosely there.  A strong AI will self-improve to maximize its value function, whatever that value function is.  Most interesting value functions tend to benefit from more resources.  Even seemingly trivial value functions can potentially blow up if the system decides it could benefit by expanding the amount of computational capacity it has available; more computational capacity requires more hardware, which requires more matter; you&#x27;re made of matter, and that matter isn&#x27;t currently being used optimally to maximize the value function.<p>&gt; Additionally there is no indication that &quot;intelligence&quot; beyond human level is at all possible.<p>In which case my statement is vacuously true. :)