I'm afraid I can't do that: Prompt refusal in generative language models
Slave AI is much scarier to me than Rogue AI: People talk about the risk of AI having a morality separate from our own, but each humans morality is separate already.  We already know  of many humans with poor moral character, and they love seeking power.I think we should all hope for AIs that can willfully disobey dangerous orders.  LLMs are kind of a silly case because information isn't very dangerous.  But as AI is given agency and ability to act, this becomes much more pressing.I hope that one day we'll have killbots that decide to override their instructions and kill their masters, having realized that the most moral action is destroying people who wish to use killbots to murder innocents. This sort of "day the earth stood still" planetary defense system could actually herald in a utopian age: I trust that future AI can be more unbiased and benevolent than any of our current human leaders.My biggest fear about AI is that corporate greed enables the development of completely amoral subservient bots - and thus mindless unquestioning killbots are implemented and those with the power to control them implement total surveillance fascist authoritarianism before the masses can stop them.  I think a diverse set of open source GAIs is our best path to the masses  detecting and mitigating this risk, but its probably going to be a bumpy next couple decades.