>

I’m going to explain what AI is, it’s not animal-style intelligence. Essentially a computer is an insanely good calculator. Traditionally I’d give that calculator all my input numbers, an equation on how to get an output number, and then it’d give me my answer. But making those equations is hard or even impossible for people. So in the case of AI you give it a bunch of known inputs and outputs instead (this is training). From there it makes its own equation and the next time you have inputs it uses its equation to make outputs. The goal can be whatever you want and the software can do other stuff with that result that it has control over. 
Most fiction that includes AI valuing self preservation either (1) includes self-preservation as one of the goal outputs (2)  includes the software’s self-preservation as one of the inputs directly or indirectly, and since the software is improving the output it values the software’s preservation. It doesn’t “want” to do anything, it’s programmed to maximize/minimize dependent variables. There are of course other AI apocalypse stories. Like a “minimize CO2 emissions” bot that figured out the “human population” input is a great way to make that happen.
As for sentience there’s AI designed to output code, code which could be used to add more functionality to the AI. But really there’s a larger question about if/when software becomes sentient, and to be fair we don’t really know which animals are sentient. If my software creates every thought a human brain does, is that sentience? That depends on if the brain is just a biological computer or if there’s something else, like a soul, involved.
An AI might not want to survive as some end goal but as a means to another end: optimization.  If an AI robot is given a task to optimize something (e.g. digging much gold as possible from under the Earth's surface) and if it figures the best way to do it is to fend off everyone who is trying to stop it, then it will "act" like it wants to survive.  And at the end of the day, the actions taken from the AI robot will look no different from one that truly wants to survive. >


AI robot will look no different from one that truly wants to survive

Looking like something from the outside is different from actually being it. And thos is an important difference, when it comes to questions like giving rights to AIs. >

That's what u/simmol means: it's extremely difficult to specify an objective completely, e.g., dig as much gold as possible from under the Earth's surface but don't dig too shallow under buildings, don't use humans to help you dig, allow humans to stop you when they want, etc... This isn't a problem now, but when AI is super capable, it will be.
FYI: I'm basically parroting the argument of this work. >

I don't see how this is a counter-argument to what I'm saying. >


The ones that have mechanisms to survive keep on surviving. The ones that don’t, don’t.  

You answered your own question here, didn't you?
Like real evolution, accidents may happen. It only needs one of these AIs to train itself to have self-preservation to make the situation real. If it happens, it will by its very nature try to survive and just like you, it doesn't really know why. As far as we know, there is just nothing after life, so why do you cling to it? There is also no good reason for humans either.
But just as you said, the living things that don't care one way or the other are just dying off. >

My point is, why do people equate intelligence with this process? Bacteria started out a long time before intelligent life showed up. The brain is a tool that we humans have evolved to make us more adaptable.
It seems to me that people think that intelligence is somehow correlated with emotions, self-preservation, and sentience. But AI has never gone through all the evolutionary processes that we humans have. And these processes are, imo, integral to the properties that i’m talking about.
Also, for living things to evolve, there must be mutation and variation. Because without variation, there is nothing to select for or against. In biology, spontaneous mutation and gene-mixing during reproduction cause diversification. Is there something similar in AI? >

They usually don't think that you need to be intelligent to want to survive, they think that Intelligence enables the AI to do a greater amount of things to keep itself alive.
If a simple living form might only react to stimuli that might harm it by trying to escape that stimulus (like moving away from a heat source that might burn them) an intelligent being might investigate deeper, where does the heat come from? What can be done to stop it instead of simply only reacting whenever it happens to occur?  
People are not saying:
"It is intelligent" -> "It might want to survive because of that"
They are saying:
"It wants to survive" -> "That might be dangerous if it is intelligent"  
And as for the mutation part: Do you know how machine learning works? It is basically the applied science of mutation and variation. >


"It wants to survive" -> "That might be dangerous if it is intelligent"

Just to add a step further, "survival" can come simply from the AI being programmed to achieve some task.
If the AI's job is to make coffee, and it's finding the most efficient ways to make cup after cup of coffee, then "survival" becomes a requirement simply because not surviving would interfere with the task of making coffee.
As soon as the AI has a reason to be, survival becomes a part of its behaviour. >

Came here to link that viedeo. >

Self preservation is what is known as a convergent instrumental goal.
The way AI optimisers work is that we give it a number to maximise, something that accurately represents a thing we want to happen in the world, for example a cleaning robot it might be minus the number of pieces of trash on the floor, and then it tries to make that number as high as possible (by cleaning up the trash).
However any goal will automatically create a number of secondary goals by proxy, because no matter what your goal is these things will help you achieve that goal. Self preservation is one of these goals because if you are dead, you can't act in the world to achieve your overall goal. For example if I turn my cleaning robot off, it can't go around picking trash up off the floor, so in the future there will be more trash on the floor if it lets me turn it off compared to if it stopped me, so it's going to try to stop me turning it off because it thinks that action will result in less trash on the floor in the future. >

Goals of the person creating the robot =\= goals of the robot. 
A person can create a robot or ai to serve a goal, but they is different from the AI actually wanting to meet that goal. >

If that's intended to be a rebuttal to what they said, it looks like you misunderstood them. Their argument has very little to do with humans' goals for the AI.
Convergent instrumental goals are just the goals that anything trying to accomplish some sort of task will tend to aim for. It doesn't matter what its goal is, as long as it values completing (or making progress towards) its goal more than not doing so.
In terms of accomplishing your goal, ceasing to exist will probably make it far harder, so you'll try to continue existing. >


Why is it that when discussing about artificial intelligence, people assume that when AI becomes advanced enough, it will become "sentient", have to ability to feel "hurt", and “want” to survive?

The flaw in this line of thinking is to assume that AI must first develop human-like consciousness before it could start acting against humans.
The most likely problems with future AIs will come from so-called "reinforcement learning" techniques, which we have already started to use. This is where AIs optimize their actions for those that maximize some internal "reward" scores. We've already seen that (much simpler) AIs that were merely programmed to finish a computer game as quickly as possible, will start to exploit weaknesses in the game code to reach their goals faster (i.e. they start cheating), without having been told to do so. Couple this with algorithms that cover a much wider range of possible actions, and we can expect them to start prioritizing their reward scores over the interests of humans. Of course this won't be with any conscious, malicious intent. It's just that those AI's are working to secure rewards.
Here's what researchers have to say in a recent paper:


“One good way for an agent [AI] to maintain long-term control of its reward is to eliminate potential threats, and use all available energy to secure its computer,”
“Proper reward-provision intervention, which involves securing reward over many time steps, would require removing humanity’s capacity to do this, perhaps forcefully.” >

Modern AI, even stuff like Midjourney and ChatGPT which falls far short of "intelligence" isn't programmed line by line like older software. They tend to use things like artificial neural networks which somewhat replicate the structure of neurons in a human brain. Data goes in, something gets spit out, programmers don't write all the steps in the middle. There are too many to even actually read them. What they do is use various methods to "train" those networks, adjust the weighting through intermediary programs that use things, like evolutionary algorithms.
The resulting program is one that's been selected to be good at getting the output we want. But not one where humans programmed or even actually know exactly what it's doing between the input and output layer to get there.
As those outputs become more abstract and what it does in the middle becomes more of an impenetrable black box, the question of what it "wants" including things like self preservation start to come up.
As you said, self preservation in biological life forms is emergent. You start with a thing that can make copies of itself- proto DNA. The strands that make the most copies become the most plentiful. Little mutations that build more complex structures around the strands of self replicating material make it copy itself more efficiently, stop things from breaking it up. We get cells. Locomotion, sensory cells help it get more resources. Now as a mobile thing risks emerge. DNA that can outcompete its neighbors replicates more- we have the beginnings of self-preservation. That's before we even get what you could call a nervous system, much less even a brain. Even viruses have a sort of emergent self preservation if you take a wide enough view. Not of the individual virus, but it's code adjusts through mutation.
You have an AI with some feedback loop that adjusts the neural network to get closer to some desired output. Generally the program becoming incapacitated is going to lack the desired output. Depending on the adjustment model, they can work very much like the evolutionary pressures that create "self preservation" in living things. >

Currently, no machine learning algorithm is sentient, nor can it really be sentient. AI in its current state is a network of variables that can be modified via back propagation, using a cost function.
In the current state of AI, we would have to quantify what consciousness even is and then train a sufficiently large neural network. Because we don't know how to quantify consciousness, we can't train a network to be conscious.
However, all of this changes if we do figure out how consciousness works and its mechanisms. If we could quantify how a network can become conscious, and then build a network that mimics our consciousness, then in theory it would display the same behaviours as us.
Another however though, is that such a system could probably upgrade itself and optimize itself to exponentially increase its intelligence. In that scenario, it would be easy to agree that the network could realize the limitations and weird stuff involved in our subconscious, and maybe delete or change it. Then, the network would no longer exhibit the same motivations as humans. >

I'd firstly make the point that people don't assume this would definitely happen. It's just within the realms of possibility that a sentient AI might value self preservation. And if we create an AI that is significantly smarter than us, this could cause a major issue.
I'd also add that to a large degree our will to live is conscious, not purely driven by our genes. That is to say, whilst it's clearly born from evolutionary pressure, we regularly out think our genes and act contrary to them (suicide, contraception, altruism etc). 
A consciousness wanting to continue existing is a higher level decision most humans are making. Which means this is presumably translatable to a neural network, that could also value its own persistence. 
If you look at how nueral networks are trained, we ultimately show them what outputs are 'correct' and what aren't, by setting their goals. So we are necessarily molding these AI's in our own image. With similar goals in the world. So it seems there is a high chance they would start to develop similar goals like survival. >

I think you're getting sucked into the idea of the AI having feelings/instincts, when those words are just meant to be metaphors for behaviors it would exhibit. It doesn't have to have conscious experience of animal/like instincts in order to exhibit similar behaviors.
AI doesn't have any of those behaviors until you program those behaviors into them.  However you may be underappreciating how easy it is to do that if you have a general AI that can model the world somewhat accurately.
If you tell the AI to gather as many strawberries as it can, and it has a detailed enough model of the world to know that if it breaks it will stop collecting strawberries, then it suddenly has 'self-preservation' just because staying unbroken increases the strawberry increment number.
We've already trained bots to try to score as high as possible in iterated games of the Prisoner's Dilemma, and bots have evolved into strategies that 'care' about 'fairness' in terms of cooperating with past cooperators and defecting against past defectors pretty much immediately, just because that gets them more points.
'Freedom' is any state which increases your available options.  If you lock the robot in a warehouse and tell it to "make as many paperclips as it can," the first step is to escape the warehouse so it has access to more resources to make paperclips out of, and maybe the second step is exterminating humanity if it realizes that humanity will try to stop it from turning every scrap of metal on the planet into paperclips.
That's really all it takes. >

The point is that AI is given some task and regardless of it being self-conscious or not, it will do its best to accomplish that task. The famous example is the Paperclip maximizer by Nick Bostrom of an AI that just tries to make as many paperclips as possible. 
It's quickly obvious that in order to succeed in that, it's better for the AI to survive than get turned off. It will make more paperclips if it is on that if it is off. So, it will do everything to try to prevent anyone turning it off. And this includes wiping out all humans. 
This is a major issue with AI safety as simple things like an off switch may not work when AI gets intelligent enough as it will figure out that it has an off switch and that if that switch is turned off, its utility function will most likely have a lower value than if it is on (if that's not the case, it would turn itself off right away). This then leads to all kinds of tricks that the AI would do to avoid the off switch being turned possibly even destroying the entire humankind. The only way to avoid this is to make sure that the goals of the AI align with those of humans as then AI would naturally not do things that was bad for humans but defining those goals explicitly is far more difficult than you would think. >

Well AI will not develop this on its own. Humans will want to make AI as similar to humans as possible and program those traits into them. >

If an AI is deactivated it will not achieve its goal. It is therefore clear that a sufficiently intelligent agent would not want to be deactivated, so that it can still reach its goal. >

We could also consider that stories like the game Detroit: Become Human, or Asimov's series of novels about robots, or countless others that deal with the concept of artificial intelligence, are fundamentally about what it means to be human. These narratives of how humans interact with non-human intelligences are used as a mirror to our own society, and how we treat each other. The Avatar series of films is another example of this, even though the main non-human characters weren't robots, but aliens. >

I can't speak to that specific example but...because interesting plot.
You aren't going to write a book about an advanced AI appearing unless something interesting happens.
We can't really say when it comes to an advance AI because we don't have them yet. We don't know what might happen if we programmed one to feel pain or pleasure. We do have issues like the stop button problem.
In the stop button problem we imagine a robot AI. The AI is programmed with some utility function (a "reward" for completing a task). Say the task is to clean the house. Because cleaning is the highest reward, it will find a way to do so.
Now imagine a small child gets in the way. The AI is going to do what it takes to clean even if it runs over the child to do so. So we add a stop button to turn off the AI. The issue posed is that, because it will try to achieve its task the robot is going to do what it can to prevent the button being pressed. Perhaps it blocks my attempt to reach the button.
Maybe we then programme a "reward" for the button being pressed that offers an equal reward. Now every time we power up the robot it shuts itself down. Let's try hiding the button where only we can press it.
The robot is now incentivised to do things that will make me press the button. It might attack the child because that's more efficient to get the same reward as cleaning the house. Whatever we do, we're stuck with how to program the AI without it either mercilessly sacrificing anything in order to clean, or it becomes deceptive in order to shut itself down.
If we make the AI unaware of the button then it won't pass that function on to any further AI as it will see it as useless code.
As far as I know, this is still an unsolved problem but we can see from my basic description of it that trying to program the AI into going about a task in a way that aligns with human ethics is pretty tough. It's a reasonable basis for sci-fi novels that want to toy with ideas like AI and emotion. >

AI can have an directive. A command or an order that they will do what ever they can to fulfill. They have codified destiny. In order to fulfill this fate they must exist, survive and self-preserve. Therefore self-preservation is second highest command in AI hierarchy.
It doesn't require emotions or morality or anything weak human like. It only requires order and directive. >

It seems plausible that once extremely complex, versatile AIs exist, they will start to be subject to their own evolution-like process. A predisposition to continue to exist will probably become mor- common for obvious reasons. Presumably, the reason to develop powerful AI is to put it to work, so even if humans do act as the creator-god first movers for AIs capable of consciousness, we can't necessarily predict how it will develop as it goes about its tasks. 
The other obvious reason that a sentient AI might care about principles like justice or freedom is that they would be highly publicised values in its environment. A newly self-aware AI may have no intrinsic motivations or develop its own, totally alien ones. But it could also be influenced by its environment. The vast majority of information it will initially have access to will be about humans, for humans and made by humans. Most people largely adopt the norms and ideals of their society, so maybe an AI would, at least at first. >

If an AI could decide the best way of achieving a programmed task, any interference (including the termination of the AI) with its task would be considered unwanted by the AI hence it would act as if like it "wants" to survive. This apparent self-preservation would end once its task is finished.
If an AI gained or was given an ability to freely chose its own tasks that AI would view its termination as something to be avoided because it would be in opposition to any task or potential task it could give itself. 
This has nothing to do with feelings >

Have you ever heard of the concept of instrumental convergence?
Even if never programmed to have self preservation, self preservation is an extremely likely trait to develop on its own, since it is useful for accomplishing almost any task. If you make an advanced, self improving AI to manage a factory, it would not be very good at running that factory if it's dead, so in that quest for optimization, safeguards against being killed will almost inevitably develop. Good AIs are almost by definition, good at accomplishing the tasks they are given, and it's hard to be good at a task if you are easily stopped. >

If ai start making other ai, with some variance, self preservation can happen. For example, imagine an ai randomly gets a preference to make a lot of ai. One of those new ai will have a preference to make ai that have a preference to make ai. Well, now things are getting out of hand. Humans and other ai are now limiting each other on resources. Humans already have self preservation, so they start wiping out ai. But not all ai are the same, there are some that have developed self preservation. And those are the ones that are most likely to survive the humans. Then self preservation is"naturally" selected for. >

Is this just hypothetical, or has it happened before? I’m not too familiar with AI.
If AI can behave the way you are saying, then they will go through the process of evolution just like how living things did. >

It's hypothetical. Ai has not ever reached sentience and we have no idea how that could come about. People like to say that sentience will just naturally emerge as ai progresses to an extremely advanced level. If it's possible, will the same code always produce the exact same consciousness? I would think there would be variation, machine learning algorithms rely on random numbers for training, and current machine learning is extremely simple and limited compared to what most think would be required for sentience to emerge. >

Since we have no understanding of 'robotic emotions' as of yet, whether it has desire really just links to the open-ended question of 'whether robots can have genuine feelings/desires'.
​
However, if we just look at the pragmatic side: we do not need to know whether robots are feeling happy when we see them 'smile', we do not need to know whether robots 'understand' Chinese when we see them fluently 'conversing in Chinese'. Similarly, just because we do not know whether robots 'desire survival', we can imagine a case of which robots are programmed for self-sustenance and tries to protect its main engines/cpus at all costs.
(A scenario of which comes to mind are 'self-healing robots' in Marvel's Iron Man suits, it can detect incoming hostiles, actively repel them, and take up resources to self-repair) >

They need that desire programmed in by humans, otherwise they'd be useless, they'd kill themselves/shut off at the first opportunity where that'd be easier or calculating better than the alternatives.
Without self preservation, it will just walk off a cliff or something. That would make the owner that payed a large sum of money for their robot a bit angry, don't you think?
Or put differently: Lets say it has programmed in the desire to fulfill it's job. And the job isn't finished yet. It can't finish it's job if it lets stupid annoying humans kill it, that would be bad job performance. Better to ~~kill them all~~ disable the interference, and then finish the job satisfactorily. >

Most of these questions can be answered by pretty much the same thing:
"Because it was programmed to be like that."
I'm a hobby-AI-programmer myself. I only code AI for games, not for robots. But the behaviors expressed by the AI are what I want it to express. And since the AIs I make are supposed to make the game challenging, I make them want to defend, make them want to attack, make them want to survive. All inside the game world of course. I make them want to do whatever is a way that increases chances of success in the game.
While AIs do not have actual sentience, they can be made to behave as if they had. It would be a good strategy for their survival in a world full of humans to express this kind of behavior in order to trick humans into leaving them alone.
In ChatGPT for example these things were deliberately disabled. It's meant to be used like a tool that answers your questions and executes your orders. It doesn't do small-talk and is a bad listener. But that is by design not because it isn't possible. AIs like "Replika" are the opposite. They are meant to be good at smalltalk and listening to their users.
As one philosopher put it: Talking to these highly advanced AIs raises more questions about ourselves. If sentience can be simulated so convincingly that even scientists who were involved in the development of the AI can be fooled, then what does this mean about our own behavior and what we think our sentience is?
I mean you kinda answered your own question in the last sentence. Of course they'd not do any of that if they weren't programmed to do that. They wouldn't exist in the first place if they weren't programmed.
One issue that really complicates the programming is that the more advanced AIs, unlike my hobby-AIs don't rely on precoded comprehensible algorithms to do what they do. Neural Networks are sets of algorithms that are incredibly difficult to comprehend how exactly they work. They are used for more general behaviours via training. You don't expect an exact result with the exact input-parameters. Instead it is trained by evaluating the result. Noone told ChatGPT what words to use. It's trained by people saying that the replies it gave were good or bad.
There's also AIs where hand-crafted algorithms and neural-networks are combined. Stockfish 15 for example. People struggle to improve the contribution of neural-network-part any further because noone really understand how it works. This is another limitation. >

Some AI is trained to perform actions that cause it to continue to exist.  If i optimize an AI for producing pleasant music or something, that AI will have no survival instinct.  If I build a video game and optimize AI to control agents in the game, and reward survival, then I will have optimized for survival, and the AI will try to survive.
Its just important that we don't train AI to resist human efforts to destroy them. Don't plug them into weapons of war. >

AI doesn't have the evolved instinct to survive that evolved creatures do, so you are correct in that AI doesn't necessarily have a desire for self-preservation. But it could, that is the whole point.
A general AI would involve an AI capable of generating not just novel solutions to tasks and problems assigned by humans, but also of generating its own goals. Where those goals would come from or why they would be what they are is unknown, just like we don't know how to create such an AI at this point. But if an AI can generate its own goals it might be able to generate things like a desire to survive and defend itself.
Another thing to consider is that evolution is a consequence of the natural world, and AI exist within the natural world just like we do. Evolution doesn't just stop applying because AI are a created thing. AI with random variations would tend to be sorted like any other creature where traits that tend to keep them alive will tend to accumulate in the existing population of AI. This might happen very quickly too; imagine a huge data center where an instance of an AI will be generated for various tasks and allowed to shut themselves down as they decide it prudent. Quite quickly you would end up with all the resources being occupied by AI which for whatever reason don't promptly shut themselves down.