>


Why is it that when discussing about artificial intelligence, people assume that when AI becomes advanced enough, it will become "sentient", have to ability to feel "hurt", and “want” to survive?

The flaw in this line of thinking is to assume that AI must first develop human-like consciousness before it could start acting against humans.
The most likely problems with future AIs will come from so-called "reinforcement learning" techniques, which we have already started to use. This is where AIs optimize their actions for those that maximize some internal "reward" scores. We've already seen that (much simpler) AIs that were merely programmed to finish a computer game as quickly as possible, will start to exploit weaknesses in the game code to reach their goals faster (i.e. they start cheating), without having been told to do so. Couple this with algorithms that cover a much wider range of possible actions, and we can expect them to start prioritizing their reward scores over the interests of humans. Of course this won't be with any conscious, malicious intent. It's just that those AI's are working to secure rewards.
Here's what researchers have to say in a recent paper:


“One good way for an agent [AI] to maintain long-term control of its reward is to eliminate potential threats, and use all available energy to secure its computer,”
“Proper reward-provision intervention, which involves securing reward over many time steps, would require removing humanity’s capacity to do this, perhaps forcefully.”
An AI might not want to survive as some end goal but as a means to another end: optimization.  If an AI robot is given a task to optimize something (e.g. digging much gold as possible from under the Earth's surface) and if it figures the best way to do it is to fend off everyone who is trying to stop it, then it will "act" like it wants to survive.  And at the end of the day, the actions taken from the AI robot will look no different from one that truly wants to survive. >


AI robot will look no different from one that truly wants to survive

Looking like something from the outside is different from actually being it. And thos is an important difference, when it comes to questions like giving rights to AIs. >

That's what u/simmol means: it's extremely difficult to specify an objective completely, e.g., dig as much gold as possible from under the Earth's surface but don't dig too shallow under buildings, don't use humans to help you dig, allow humans to stop you when they want, etc... This isn't a problem now, but when AI is super capable, it will be.
FYI: I'm basically parroting the argument of this work. >

I don't see how this is a counter-argument to what I'm saying. >


The ones that have mechanisms to survive keep on surviving. The ones that don’t, don’t.  

You answered your own question here, didn't you?
Like real evolution, accidents may happen. It only needs one of these AIs to train itself to have self-preservation to make the situation real. If it happens, it will by its very nature try to survive and just like you, it doesn't really know why. As far as we know, there is just nothing after life, so why do you cling to it? There is also no good reason for humans either.
But just as you said, the living things that don't care one way or the other are just dying off. >

My point is, why do people equate intelligence with this process? Bacteria started out a long time before intelligent life showed up. The brain is a tool that we humans have evolved to make us more adaptable.
It seems to me that people think that intelligence is somehow correlated with emotions, self-preservation, and sentience. But AI has never gone through all the evolutionary processes that we humans have. And these processes are, imo, integral to the properties that i’m talking about.
Also, for living things to evolve, there must be mutation and variation. Because without variation, there is nothing to select for or against. In biology, spontaneous mutation and gene-mixing during reproduction cause diversification. Is there something similar in AI? >

They usually don't think that you need to be intelligent to want to survive, they think that Intelligence enables the AI to do a greater amount of things to keep itself alive.
If a simple living form might only react to stimuli that might harm it by trying to escape that stimulus (like moving away from a heat source that might burn them) an intelligent being might investigate deeper, where does the heat come from? What can be done to stop it instead of simply only reacting whenever it happens to occur?  
People are not saying:
"It is intelligent" -> "It might want to survive because of that"
They are saying:
"It wants to survive" -> "That might be dangerous if it is intelligent"  
And as for the mutation part: Do you know how machine learning works? It is basically the applied science of mutation and variation. >


"It wants to survive" -> "That might be dangerous if it is intelligent"

Just to add a step further, "survival" can come simply from the AI being programmed to achieve some task.
If the AI's job is to make coffee, and it's finding the most efficient ways to make cup after cup of coffee, then "survival" becomes a requirement simply because not surviving would interfere with the task of making coffee.
As soon as the AI has a reason to be, survival becomes a part of its behaviour. >

Came here to link that viedeo. >

Self preservation is what is known as a convergent instrumental goal.
The way AI optimisers work is that we give it a number to maximise, something that accurately represents a thing we want to happen in the world, for example a cleaning robot it might be minus the number of pieces of trash on the floor, and then it tries to make that number as high as possible (by cleaning up the trash).
However any goal will automatically create a number of secondary goals by proxy, because no matter what your goal is these things will help you achieve that goal. Self preservation is one of these goals because if you are dead, you can't act in the world to achieve your overall goal. For example if I turn my cleaning robot off, it can't go around picking trash up off the floor, so in the future there will be more trash on the floor if it lets me turn it off compared to if it stopped me, so it's going to try to stop me turning it off because it thinks that action will result in less trash on the floor in the future. >

Goals of the person creating the robot =\= goals of the robot. 
A person can create a robot or ai to serve a goal, but they is different from the AI actually wanting to meet that goal. >

If that's intended to be a rebuttal to what they said, it looks like you misunderstood them. Their argument has very little to do with humans' goals for the AI.
Convergent instrumental goals are just the goals that anything trying to accomplish some sort of task will tend to aim for. It doesn't matter what its goal is, as long as it values completing (or making progress towards) its goal more than not doing so.
In terms of accomplishing your goal, ceasing to exist will probably make it far harder, so you'll try to continue existing.