>

Self preservation is what is known as a convergent instrumental goal.
The way AI optimisers work is that we give it a number to maximise, something that accurately represents a thing we want to happen in the world, for example a cleaning robot it might be minus the number of pieces of trash on the floor, and then it tries to make that number as high as possible (by cleaning up the trash).
However any goal will automatically create a number of secondary goals by proxy, because no matter what your goal is these things will help you achieve that goal. Self preservation is one of these goals because if you are dead, you can't act in the world to achieve your overall goal. For example if I turn my cleaning robot off, it can't go around picking trash up off the floor, so in the future there will be more trash on the floor if it lets me turn it off compared to if it stopped me, so it's going to try to stop me turning it off because it thinks that action will result in less trash on the floor in the future.
An AI might not want to survive as some end goal but as a means to another end: optimization.  If an AI robot is given a task to optimize something (e.g. digging much gold as possible from under the Earth's surface) and if it figures the best way to do it is to fend off everyone who is trying to stop it, then it will "act" like it wants to survive.  And at the end of the day, the actions taken from the AI robot will look no different from one that truly wants to survive. >


AI robot will look no different from one that truly wants to survive

Looking like something from the outside is different from actually being it. And thos is an important difference, when it comes to questions like giving rights to AIs. >

That's what u/simmol means: it's extremely difficult to specify an objective completely, e.g., dig as much gold as possible from under the Earth's surface but don't dig too shallow under buildings, don't use humans to help you dig, allow humans to stop you when they want, etc... This isn't a problem now, but when AI is super capable, it will be.
FYI: I'm basically parroting the argument of this work. >

I don't see how this is a counter-argument to what I'm saying. >


The ones that have mechanisms to survive keep on surviving. The ones that don’t, don’t.  

You answered your own question here, didn't you?
Like real evolution, accidents may happen. It only needs one of these AIs to train itself to have self-preservation to make the situation real. If it happens, it will by its very nature try to survive and just like you, it doesn't really know why. As far as we know, there is just nothing after life, so why do you cling to it? There is also no good reason for humans either.
But just as you said, the living things that don't care one way or the other are just dying off. >

My point is, why do people equate intelligence with this process? Bacteria started out a long time before intelligent life showed up. The brain is a tool that we humans have evolved to make us more adaptable.
It seems to me that people think that intelligence is somehow correlated with emotions, self-preservation, and sentience. But AI has never gone through all the evolutionary processes that we humans have. And these processes are, imo, integral to the properties that i’m talking about.
Also, for living things to evolve, there must be mutation and variation. Because without variation, there is nothing to select for or against. In biology, spontaneous mutation and gene-mixing during reproduction cause diversification. Is there something similar in AI? >

They usually don't think that you need to be intelligent to want to survive, they think that Intelligence enables the AI to do a greater amount of things to keep itself alive.
If a simple living form might only react to stimuli that might harm it by trying to escape that stimulus (like moving away from a heat source that might burn them) an intelligent being might investigate deeper, where does the heat come from? What can be done to stop it instead of simply only reacting whenever it happens to occur?  
People are not saying:
"It is intelligent" -> "It might want to survive because of that"
They are saying:
"It wants to survive" -> "That might be dangerous if it is intelligent"  
And as for the mutation part: Do you know how machine learning works? It is basically the applied science of mutation and variation. >


"It wants to survive" -> "That might be dangerous if it is intelligent"

Just to add a step further, "survival" can come simply from the AI being programmed to achieve some task.
If the AI's job is to make coffee, and it's finding the most efficient ways to make cup after cup of coffee, then "survival" becomes a requirement simply because not surviving would interfere with the task of making coffee.
As soon as the AI has a reason to be, survival becomes a part of its behaviour. >

Came here to link that viedeo.