Answers.Body: <p>This is an interesting question actually.</p>
  
  <p>There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee.</p>
  
  <p>It's based on such statements: </p>
  
  <ul>
  <li><p>Mind creates its own model of the world it exists in.</p></li>
  <li><p>It makes predictions about everything all the time (actually Jeff Hawkins states that this is the main characteristic of intelligence).</p></li>
  <li><p>When prediction about something wasn't followed by appropriate behavior of the world, then this thing gets very interesting to the mind (the model is wrong and should be corrected) and needs more attention.</p></li>
  </ul>
  
  <p>For example, when you look at left human eye your brain predicts that it's a human face and there should be second eye to the right. You look to the right and see a.. nose! What a surprise! It now takes all your attention and you have this motivation to make more observations about such a strange thing that did not fit into your model.</p>
  
  <p>So I'd say that AI might do something certain according to its model or behave randomly while the predictions it is making about the world are true. But once some prediction is broken the AI gets a motivation to do error-correction to its model.</p>
  
  <p>In a simple case a machine starts at a total randomness just doing everything it can with its output. While it has no model or a random model when it detects some kind of order or repeated patterns it is getting "interested" and adds it to the model. After a while, the model becomes more sophisticated making more complex predictions and detecting higher level mistakes in a model. Slowly it gets to know what to do to observe something interesting to it, instead of just remembering everything.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-15T09:38:01.063Z
Answers.Text: Thank you for the contribution! I have come to basically the same conclusions... now thinking of a way to implement it :)
Answers.UserDisplayName: Aleksei Maide
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T09:21:16.853Z
Answers.Text: This answer makes an important point. Error correction on prediction models would provide a great incentive for an intelligent AI to learn and act in curious manner.
Answers.UserDisplayName: Seth Simba
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-15T06:40:55.073Z
Answers.LastActivityDate: 2017-09-15T06:40:55.073Z
Answers.UserDisplayName: Ivan Bogush
Answers.Body: <p>I asked <a href="https://www.ualberta.ca/science/about-us/contact-us/faculty-directory/rich-sutton" rel="nofollow noreferrer">professor Richard Sutton</a> a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine motivation seems to me like a dedicated field of research.</p>
  
  <p>Typically, machines are motivated by what we call an <em>objective function</em> or a <em>cost function</em> or a <em>loss function</em>. These are different names for the same concept. Sometimes, they are denoted by </p>
  
  <p><span class="math-container">$$L(a)$$</span></p>
  
  <p>The goal of the machine is then to solve either a minimization problem, <span class="math-container">$\min_a L(a)$</span>, or a maximization problem, <span class="math-container">$\max_a L(a)$</span>, depending on the definition of <span class="math-container">$L$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-08-27T17:51:34.497Z
Answers.LastActivityDate: 2019-10-07T19:22:56.667Z
Answers.LastEditDate: 2019-10-07T19:22:56.667Z
Answers.UserDisplayName: A.Rashad
Answers.Body: <p>I think we give ourselves too much credit by already referring to our algorithms and machines as actually thinking and acting on motivations. In my opinion we still have a bit to go before we can actually refer to a human creation as thinking or being able to have motivations more then basic physical ones.</p>
  <p>By that I would say that a Machines' or AI algorithms' motivations are similar to a car engine. Simple and basic, the &quot;motivations&quot; of a car engine to run are just the first and second laws of thermodynamics, namely the conservation of energy and the exchange between energy types, and the always increasing level of entropy in a closed system.</p>
  <p>By having a really specific design, we can insert fuel in the system and create a lot of potential energy which will &quot;motivate&quot; the engine to transform it in other types of energy (heat, sound, etc.)</p>
  <p>An AI algorithm is exactly the same, it's just that now we're playing with electricity. By putting multiple levels of abstractions from the actual level of electrons moving through wire, up to your python Deep Learning algorithm training to learn how to recognize images of dogs. The concept is similar in my opinion, for now we do not have machines that are complex enough to have higher-level motivations, or even develop them by themselves.</p>
  <p>As the other answers pointed out, specific algorithms, namely reinforcement learning try to emulate those &quot;needs&quot; and &quot;motivations&quot;, but in the end in my opinion, for now, they are still just emulations. Similar to other Deep learning algorithms, the same basic concept described at the beginning applies, trying to minimize the error by emulating different concepts that we know, as conservation of energy, following the path of least resistance, maintaining the laws of entropy, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-22T16:45:13.120Z
Answers.LastActivityDate: 2021-04-22T16:45:13.120Z
Answers.UserDisplayName: Darius Chira
Answers.Body: <p>I've spent some time thinking about this in the context of games.</p>
  
  <p>The problem with reward functions is that they generally involve weighting nodes, which is useful but ultimately materially meaningless.</p>
  
  <p>Here are two materially meaningful rewards:</p>
  
  <p><strong>COMPUTATIONAL RESOURCES</strong></p>
  
  <p>Consider a game where an AI is competing not for points, but for processor time and memory.</p>
  
  <p>The better the algorithm performs at the game, the more memory and processing it has access to.  This has a practical effect--the more resources available to the automata, the stronger its capabilities.  (i.e. it's <a href="https://en.wikipedia.org/wiki/Bounded_rationality" rel="nofollow noreferrer">rationality is less bounded</a> in terms of time and space to make a decision.)  Thus the algorithm would be "motivated" to prevail such a contest.</p>
  
  <p><strong>ENERGY</strong></p>
  
  <p>Any automata with a sufficient degree of "self awareness", here specifically referring to the knowledge that it requires energy to process, would be motivated to self-optimize its own code to eliminate unnecessary flipping of bits (unnecessary energy consumption.)</p>
  
  <p>Such an algorithm would also be motivated to ensure its power supply so that it can continue to function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-07T20:10:57.187Z
Answers.LastActivityDate: 2019-10-07T20:10:57.187Z
Answers.UserDisplayName: DukeZhou
Answers.Body: <p>Good verses bad?  From an imaging standpoint, humans/biologicals tend to notice details not “imagined”.  A “good” feedback learning loop might be refining the correctness of the imagined. Using efforts to reduce the number or amount discrepancies between imagined or predicted environment and perceived reality as measured against “beneficial” outcomes.  I’m not sure what a Ai wood think of as good/bad in the abstract, in my limited considerations of imagination in humans, I seem most useful in recognizing things that be mor meaning than the general background.  As the imagination becomes more able to predict a more “real” outcome, it is more useful to the individual in predicting a more correct outcome and greater usefulness in defining things that deverge form the expected.  Sort simplistic I realize, but intelligence might be as simple as refining.the ability to imagine a result and judging the more meaningful of the descrpencies .</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-16T12:13:08.337Z
Answers.Text: Good/Bad are entirely human concepts, humanity lived for millions of years without those, so there may not be useful words in evaluating... "urge for action", these concepts are a higher level of abstraction
Answers.UserDisplayName: Aleksei Maide
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-14T00:19:49.807Z
Answers.LastActivityDate: 2021-11-14T16:46:37.267Z
Answers.LastEditDate: 2021-11-14T16:46:37.267Z
Answers.UserDisplayName: Chad Neff
Answers.Body: <p>The current method to implement motivation is some kind of artificial reward. <a href="https://deepmind.com/research/dqn/" rel="nofollow noreferrer">Deepmind's DQN</a> for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust its actions to get the most points and therefore the most reward. This is called <a href="https://en.wikipedia.org/wiki/Reinforcement_learning" rel="nofollow noreferrer">reinforcement learing</a>. The reward <strong>motivates</strong> the AI to adapt its actions, so to speak.</p>
  
  <p>In a more technical term, the AI wants to maximize utility, which depends on the implemented <a href="https://stackoverflow.com/questions/14360893/intelligent-agents-utility-function">utility function</a>. In the case of DQN, this would be maximizing the score in the game.</p>
  
  <p>The human brain functions in a similar fashion, although a little more complex and often not as straight forward. We as humans usually try to adjust our actions to produce a high output of <strong>dopamine</strong> and <strong>serotonin</strong>. This is in a way similar to the reward used to control AIs during reinforcement learning. The human brain learns which actions produce the most amount of those substances and finds strategies to maximize the output. This is, of course, a simplification of this complex process, but you get the picture.</p>
  
  <p>When you talk about motivation, please don't mix it up with <strong>consciousness</strong> or <a href="https://en.wikipedia.org/wiki/Qualia" rel="nofollow noreferrer"><strong>qualia</strong></a>. Those are not required for motivation at all. If you want to discuss consciousness and qualia in AI, that's a totally different ball game.</p>
  
  <p>A child isn't curious for the sake of curiosity. It gets positive reinforcement when exploring because the utility function of the child's brain rewards exploration by releasing rewarding neurotransmitters. So the mechanism is the same. Applying this to AI means defining a utility function that rewards new experiences. There is no inner drive without some kind of reinforcing reward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-07T20:18:08.017Z
Answers.Text: in regards to the edit i think a good example of "a utility function that rewards new experience" would be the novelty search fitness functions proposed by ken stanley to be used in his neat algorithm.
Answers.UserDisplayName: nickw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-08-27T17:53:06.690Z
Answers.LastActivityDate: 2019-10-08T02:59:34.470Z
Answers.LastEditDate: 2019-10-08T02:59:34.470Z
Answers.UserDisplayName: Demento
Body: <p>Currently, within the AI development field, the main focus seems to be on pattern recognition and machine learning. Learning is about adjusting internal variables based on a feedback loop.</p>
  <p><a href="https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs" rel="nofollow noreferrer">Maslow's hierarchy of needs</a> is a theory in psychology proposed by <em>Abraham Maslow</em> that claims that individuals' most basic needs must be met, before they become motivated to achieve higher-level needs.</p>
  <ul>
  <li>What could possibly motivate a machine to act?</li>
  <li>Should a machine have some sort of DNA-like structure that would describe its hierarchy of needs (similar to Maslow's theory)?</li>
  <li>What could be the fundamental needs of a machine?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-27T20:25:55.827Z
Comments.Text: Interesting question, and welcome to AI!  (I have a few thoughts on the subject, related to game theory, and other contributors have talked about [goal oriented learning](http://cognet.mit.edu/book/goal-driven-learning) in relation to algorithms.)
Comments.UserDisplayName: DukeZhou
Tags: <philosophy><agi><rewards>
Title: What would motivate a machine?
UserDisplayName: Aleksei Maide
ThreadText: # What would motivate a machine?
  
  (Asked by: username_0 on 2017-08-27)
  
  Currently, within the AI development field, the main focus seems to be on pattern recognition and machine learning. Learning is about adjusting internal variables based on a feedback loop.
  
  [Maslow's hierarchy of needs](https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs) is a theory in psychology proposed by *Abraham Maslow* that claims that individuals' most basic needs must be met, before they become motivated to achieve higher-level needs.
  
  * What could possibly motivate a machine to act?
  * Should a machine have some sort of DNA-like structure that would describe its hierarchy of needs (similar to Maslow's theory)?
  * What could be the fundamental needs of a machine?
  
  **Tags:** philosophy, agi, rewards
  
  > **Comments:**
  > **username_1**: Interesting question, and welcome to AI!  (I have a few thoughts on the subject, related to game theory, and other contributors have talked about [goal oriented learning](http://cognet.mit.edu/book/goal-driven-learning) in relation to algorithms.)  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_7 on 2017-08-27. Score: 5 | Accepted answer
  
  The current method to implement motivation is some kind of artificial reward. [Deepmind's DQN](https://deepmind.com/research/dqn/) for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust its actions to get the most points and therefore the most reward. This is called [reinforcement learing](https://en.wikipedia.org/wiki/Reinforcement_learning). The reward **motivates** the AI to adapt its actions, so to speak.
  
  In a more technical term, the AI wants to maximize utility, which depends on the implemented [utility function](https://stackoverflow.com/questions/14360893/intelligent-agents-utility-function). In the case of DQN, this would be maximizing the score in the game.
  
  The human brain functions in a similar fashion, although a little more complex and often not as straight forward. We as humans usually try to adjust our actions to produce a high output of **dopamine** and **serotonin**. This is in a way similar to the reward used to control AIs during reinforcement learning. The human brain learns which actions produce the most amount of those substances and finds strategies to maximize the output. This is, of course, a simplification of this complex process, but you get the picture.
  
  When you talk about motivation, please don't mix it up with **consciousness** or [**qualia**](https://en.wikipedia.org/wiki/Qualia). Those are not required for motivation at all. If you want to discuss consciousness and qualia in AI, that's a totally different ball game.
  
  A child isn't curious for the sake of curiosity. It gets positive reinforcement when exploring because the utility function of the child's brain rewards exploration by releasing rewarding neurotransmitters. So the mechanism is the same. Applying this to AI means defining a utility function that rewards new experiences. There is no inner drive without some kind of reinforcing reward.
  
  > **Comments:**
  > **username_8**: in regards to the edit i think a good example of "a utility function that rewards new experience" would be the novelty search fitness functions proposed by ken stanley to be used in his neat algorithm.  
  
  ---
  
  ### Answer by username_2 on 2017-09-15. Score: 5
  
  This is an interesting question actually.
  
  There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee.
  
  It's based on such statements:
  
  * Mind creates its own model of the world it exists in.
  * It makes predictions about everything all the time (actually Jeff Hawkins states that this is the main characteristic of intelligence).
  * When prediction about something wasn't followed by appropriate behavior of the world, then this thing gets very interesting to the mind (the model is wrong and should be corrected) and needs more attention.
  
  For example, when you look at left human eye your brain predicts that it's a human face and there should be second eye to the right. You look to the right and see a.. nose! What a surprise! It now takes all your attention and you have this motivation to make more observations about such a strange thing that did not fit into your model.
  
  So I'd say that AI might do something certain according to its model or behave randomly while the predictions it is making about the world are true. But once some prediction is broken the AI gets a motivation to do error-correction to its model.
  
  In a simple case a machine starts at a total randomness just doing everything it can with its output. While it has no model or a random model when it detects some kind of order or repeated patterns it is getting "interested" and adds it to the model. After a while, the model becomes more sophisticated making more complex predictions and detecting higher level mistakes in a model. Slowly it gets to know what to do to observe something interesting to it, instead of just remembering everything.
  
  > **Comments:**
  > **username_0**: Thank you for the contribution! I have come to basically the same conclusions... now thinking of a way to implement it :)  
  > **username_3**: This answer makes an important point. Error correction on prediction models would provide a great incentive for an intelligent AI to learn and act in curious manner.  
  
  ---
  
  ### Answer by username_4 on 2017-08-27. Score: 3
  
  I asked [professor Richard Sutton](https://www.ualberta.ca/science/about-us/contact-us/faculty-directory/rich-sutton) a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine motivation seems to me like a dedicated field of research.
  
  Typically, machines are motivated by what we call an *objective function* or a *cost function* or a *loss function*. These are different names for the same concept. Sometimes, they are denoted by
  
  $$L(a)$$
  
  The goal of the machine is then to solve either a minimization problem, $\min\_a L(a)$, or a maximization problem, $\max\_a L(a)$, depending on the definition of $L$.
  
  
  ---
  
  ### Answer by username_5 on 2021-04-22. Score: 1
  
  I think we give ourselves too much credit by already referring to our algorithms and machines as actually thinking and acting on motivations. In my opinion we still have a bit to go before we can actually refer to a human creation as thinking or being able to have motivations more then basic physical ones.
  
  By that I would say that a Machines' or AI algorithms' motivations are similar to a car engine. Simple and basic, the "motivations" of a car engine to run are just the first and second laws of thermodynamics, namely the conservation of energy and the exchange between energy types, and the always increasing level of entropy in a closed system.
  
  By having a really specific design, we can insert fuel in the system and create a lot of potential energy which will "motivate" the engine to transform it in other types of energy (heat, sound, etc.)
  
  An AI algorithm is exactly the same, it's just that now we're playing with electricity. By putting multiple levels of abstractions from the actual level of electrons moving through wire, up to your python Deep Learning algorithm training to learn how to recognize images of dogs. The concept is similar in my opinion, for now we do not have machines that are complex enough to have higher-level motivations, or even develop them by themselves.
  
  As the other answers pointed out, specific algorithms, namely reinforcement learning try to emulate those "needs" and "motivations", but in the end in my opinion, for now, they are still just emulations. Similar to other Deep learning algorithms, the same basic concept described at the beginning applies, trying to minimize the error by emulating different concepts that we know, as conservation of energy, following the path of least resistance, maintaining the laws of entropy, etc.
  
  
  ---
  
  ### Answer by username_1 on 2019-10-07. Score: 1
  
  I've spent some time thinking about this in the context of games.
  
  The problem with reward functions is that they generally involve weighting nodes, which is useful but ultimately materially meaningless.
  
  Here are two materially meaningful rewards:
  
  **COMPUTATIONAL RESOURCES**
  
  Consider a game where an AI is competing not for points, but for processor time and memory.
  
  The better the algorithm performs at the game, the more memory and processing it has access to. This has a practical effect--the more resources available to the automata, the stronger its capabilities. (i.e. it's [rationality is less bounded](https://en.wikipedia.org/wiki/Bounded_rationality) in terms of time and space to make a decision.) Thus the algorithm would be "motivated" to prevail such a contest.
  
  **ENERGY**
  
  Any automata with a sufficient degree of "self awareness", here specifically referring to the knowledge that it requires energy to process, would be motivated to self-optimize its own code to eliminate unnecessary flipping of bits (unnecessary energy consumption.)
  
  Such an algorithm would also be motivated to ensure its power supply so that it can continue to function.
  
  
  ---
  
  ### Answer by username_6 on 2021-11-14. Score: 0
  
  Good verses bad? From an imaging standpoint, humans/biologicals tend to notice details not “imagined”. A “good” feedback learning loop might be refining the correctness of the imagined. Using efforts to reduce the number or amount discrepancies between imagined or predicted environment and perceived reality as measured against “beneficial” outcomes. I’m not sure what a Ai wood think of as good/bad in the abstract, in my limited considerations of imagination in humans, I seem most useful in recognizing things that be mor meaning than the general background. As the imagination becomes more able to predict a more “real” outcome, it is more useful to the individual in predicting a more correct outcome and greater usefulness in defining things that deverge form the expected. Sort simplistic I realize, but intelligence might be as simple as refining.the ability to imagine a result and judging the more meaningful of the descrpencies .
  
  > **Comments:**
  > **username_0**: Good/Bad are entirely human concepts, humanity lived for millions of years without those, so there may not be useful words in evaluating... "urge for action", these concepts are a higher level of abstraction  
  
  ---
  
  