&quot;everyone just thinks we should rush forward into this not concerned about an AI that is super human&quot;<p>No, on the contrary, nearly everyone who spends any amount of time thinking about it quickly realizes the risks.<p>The concession is the realization that the technology is an inevitability (because of the immense power it grants the wielder, and because of the wide gradient of safe and useful AI to dangerous AI).<p>I think you would have an extremely tough time deciding where to draw the line. The closest parallel we have may be the export controls on cryptography or the ridiculousness that emerged from the AACS encryption key fiasco.