Answers.Body: <p>There are a lot of approaches you could take for this. Creating a realistic artificial analog for fear as implemented biologically in animals might be possible, but there is quite a lot involved in a real animal's fear response that would not apply in simpler AI bots available now. For instance, an animal entering a state of fear will typically use hormones to signal changes throughout its body, favouring resource expenditure and risk taking ("fight or flight").</p>
  
  <p>In basic reinforcement learning, the neural network would not need to directly decide switch on a "fear mode". Instead, you can make use of some design in the agent and learning algorithm to help learn from rare but significant events. Here are a few ideas:</p>
  
  <ul>
  <li><p>Experience replay. You may already be doing this in the Pacman scenario, if you are using DQN or something similar. Storing the state transition and reward that caused a large positive or negative reward, and repeatedly learning from it should offset your concern</p></li>
  <li><p>Prioritised sweeping. You can use larger differences experienced between predicted and actual reward to bias sampling from your replay memory towards significant events and those linked closely to them. </p></li>
  <li><p>Planning. With a predictive model - maybe based on sampled transitions (you can re-use the experience replay memory for this), or maybe a trained state transition prediction network - then you can look multiple steps ahead by simulating. There is a strong relation between RL and look-ahead planning too, they are very similar algorithm. The difference is which states and actions are being considered, and whether they are being simulated or experienced. Experience replay blurs the line here - it can be framed as learning from memory, or improving predictions for planning. Planning helps by optimising decisions without needing to repeat experiences as much - a combination of planning and learning can be far more powerful than either in isolation.</p></li>
  <li><p>Smarter exploratory action selection. Epsilon-greedy, where you either take a greedy action or take a completely random action, completely ignores how much you may have already learned about alternative actions and their relative merit. You can use something like Upper Confidence Bound with a value-based agent.</p></li>
  <li><p>In a deterministic world, increase the batch size for learning and planning, as you can trust that when a transition is learned once, that you know everything about it. </p></li>
  </ul>
  
  <p>You will need to experiment in each environment. You can make learning agents that are more conservative about exploring near low reward areas. However, if the environment is such that it is necessary to take risks in order to get to the best rewards (which is often the case in games) then it may not be optimal in terms of learning time to have a "timid" agent. For instance in your example of Pacman, sometimes the ghosts should be avoided, sometimes they should be chased. If the agent learned strong aversion initially, it might take a long time to overcome this and learn to chase them after eating a power-up.</p>
  
  <p>For your example of the spider, as the constructor of the experiment then you know that the bite is bad every time and that the agent must avoid it as much as possible. To most RL algorithms, there is no such knowledge, except gained through experience. An MDP world model does not need to match common sense, it may be that a spider bite is bad (-10 reward) 90% of the time and good 10% of the time (+1000 reward). The agent can only discover this by being bitten multiple times . . . RL typically does not start with any system to make assumptions about this sort of thing, and it is impossible to come up with a general rule about all possible MDPs. Instead, for a basic RL system, you can consider modifying hyperparameters or focusing on key events as suggested above. Outside of a basic RL system there could be merit in replicating other things, such as "instinctive" fear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T14:02:33.473Z
Answers.Text: @DuttaA: I agree, which is why I suggest a bunch of stuff that is not "real fear (tm)". I think a very basic "instinctive fear" using RL would involve somehow adding a prior low value into the value function programmatically without actual experience.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-12T20:42:04.680Z
Answers.LastActivityDate: 2018-08-12T22:07:09.533Z
Answers.LastEditDate: 2018-08-12T22:07:09.533Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Fear of this kind is an irrational response (large negative incentive in response to a small risk).  Modeling fear would need to model a "grossness" factor associated with, for example, spiders so that the normally un-proportional response would occur.  The "grossness" factor could be manifested in many other forms to magnify a response to a previously unpleasant, though not particularly dangerous, experience.  Such fear can also be inspired by hearsay (think hysteria caused by a sensational news story).
  A NN would normally only respond minimally to a minimal risk.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-05T17:06:12.033Z
Answers.LastActivityDate: 2018-10-05T17:06:12.033Z
Answers.UserDisplayName: SchroedingersCat
Answers.Body: <p>I think there are 2 ways to make this happen: 1) explicitly program fear as a constraint or parameter in some logical expression, or 2) utilize a large set of training data to teach fear.</p>
  
  <p>Think about a basic Pacman game-- whether Pacman fears the ghosts or doesn't fear them is hard to tell, but they ARE ghosts and Pacman avoids them so I think it's safe we can use this as a basic example of "fear". Since, in this game, fear = avoidance, you could logically program avoidance to be some sort of distance. I tried this with Pacman reinforcement learning. I tried to set a distance of 5 squares to the ghosts and anytime Pacman could see a ghost within 5 squares, he would move in a different direction. What I found is that while Pacman will try to avoid ghosts, he doesn't know strategy (or have intelligence). Pacman would simply move away from ghosts until he got boxed in.</p>
  
  <p>My point is that you can program your network to avoid spiders to not get bit, but without training, you will just be creating a basic parameter that might cause problems if there are 100 super aggressive spiders coming at you! The better way is to use some base logic to avoid spiders, but then train the network to be rewarded the better spiders are avoided. </p>
  
  <p>Now, there are many situations of fear so this one example with Pacman would not necessarily apply to all... Just trying to give some insight in my experience with teaching fear with reinforcement learning in Pacman.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-12T17:16:44.530Z
Answers.Text: I'm thinking that for some things like fire it needs to be an instinct. Because by the time you've had a few "training examples" you'd be pretty burned. But with other things you should only need one example like getting bit by a hamster you should learn hamsters have sharp teeth so don't put your hand in their mouth. So for some things instincts should just prevent you from doing things like putting your hand in a fire. Or make you jump back. But should you also be scared of going near fire if you learn that you might get hit by a spark?
Answers.UserDisplayName: zooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T20:16:50.910Z
Answers.Text: @zooby in my experience little kids don't really learn to fear the hot stove or fire until they do get burned.  (s'why you've got to keep such a close eye on them!)  I'd say that if the algorithm is learning avoidance through a technique like reinforcement learning, it is "learning what to fear", whereas, if it has the avoidance mechanism pre-programmed, that would be "instinct".
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-12T16:46:54.603Z
Answers.LastActivityDate: 2018-08-12T16:46:54.603Z
Answers.UserDisplayName: CaptainPlanet
Answers.Body: <p>I would suggest having the agent weight its learning from a given event based on the severity of the consequences for that event happening. Eg. Have it develop a threat model like those typically drafted up in the Information Security field. High risk but low probability is something that can be accounted for and judged against. </p>
  
  <p>Trying to directly imitate human fear would be silly, you'd likely end up with AIs that have phobias if you succeeded too well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T20:41:33.720Z
Answers.LastActivityDate: 2018-08-14T20:41:33.720Z
Answers.UserDisplayName: Christopher Griffith
Body: <p>If you've been attacked by a spider once, chances are you'll never go near a spider again.</p>
  <p>In a neural network model, having a bad experience with a spider will slightly decrease the probability you will go near a spider depending on the learning rate. This is not good.</p>
  <p>How can you program fear into a neural network, such that you don't need hundreds of examples of being bitten by a spider in order to ignore the spider (and also that it doesn't just lower the probability that you will choose to go near a spider)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-18T06:06:16.773Z
Comments.Text: WP article One-shoot learning https://en.wikipedia.org/wiki/One-shot_learning includes a section about learning with one example.
Comments.UserDisplayName: Jaume Oliver Lafont
Tags: <neural-networks><machine-learning><reinforcement-learning>
Title: How do you program fear into a neural network?
UserDisplayName: zooby
ThreadText: # How do you program fear into a neural network?
  
  (Asked by: username_0 on 2018-08-12)
  
  If you've been attacked by a spider once, chances are you'll never go near a spider again.
  
  In a neural network model, having a bad experience with a spider will slightly decrease the probability you will go near a spider depending on the learning rate. This is not good.
  
  How can you program fear into a neural network, such that you don't need hundreds of examples of being bitten by a spider in order to ignore the spider (and also that it doesn't just lower the probability that you will choose to go near a spider)?
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning
  > **Comments:**
  > **username_1**: WP article One-shoot learning https://en.wikipedia.org/wiki/One-shot_learning includes a section about learning with one example.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2018-08-12. Score: 5
  
  There are a lot of approaches you could take for this. Creating a realistic artificial analog for fear as implemented biologically in animals might be possible, but there is quite a lot involved in a real animal's fear response that would not apply in simpler AI bots available now. For instance, an animal entering a state of fear will typically use hormones to signal changes throughout its body, favouring resource expenditure and risk taking ("fight or flight").
  
  In basic reinforcement learning, the neural network would not need to directly decide switch on a "fear mode". Instead, you can make use of some design in the agent and learning algorithm to help learn from rare but significant events. Here are a few ideas:
  
  * Experience replay. You may already be doing this in the Pacman scenario, if you are using DQN or something similar. Storing the state transition and reward that caused a large positive or negative reward, and repeatedly learning from it should offset your concern
  * Prioritised sweeping. You can use larger differences experienced between predicted and actual reward to bias sampling from your replay memory towards significant events and those linked closely to them.
  * Planning. With a predictive model - maybe based on sampled transitions (you can re-use the experience replay memory for this), or maybe a trained state transition prediction network - then you can look multiple steps ahead by simulating. There is a strong relation between RL and look-ahead planning too, they are very similar algorithm. The difference is which states and actions are being considered, and whether they are being simulated or experienced. Experience replay blurs the line here - it can be framed as learning from memory, or improving predictions for planning. Planning helps by optimising decisions without needing to repeat experiences as much - a combination of planning and learning can be far more powerful than either in isolation.
  * Smarter exploratory action selection. Epsilon-greedy, where you either take a greedy action or take a completely random action, completely ignores how much you may have already learned about alternative actions and their relative merit. You can use something like Upper Confidence Bound with a value-based agent.
  * In a deterministic world, increase the batch size for learning and planning, as you can trust that when a transition is learned once, that you know everything about it.
  
  You will need to experiment in each environment. You can make learning agents that are more conservative about exploring near low reward areas. However, if the environment is such that it is necessary to take risks in order to get to the best rewards (which is often the case in games) then it may not be optimal in terms of learning time to have a "timid" agent. For instance in your example of Pacman, sometimes the ghosts should be avoided, sometimes they should be chased. If the agent learned strong aversion initially, it might take a long time to overcome this and learn to chase them after eating a power-up.
  
  For your example of the spider, as the constructor of the experiment then you know that the bite is bad every time and that the agent must avoid it as much as possible. To most RL algorithms, there is no such knowledge, except gained through experience. An MDP world model does not need to match common sense, it may be that a spider bite is bad (-10 reward) 90% of the time and good 10% of the time (+1000 reward). The agent can only discover this by being bitten multiple times . . . RL typically does not start with any system to make assumptions about this sort of thing, and it is impossible to come up with a general rule about all possible MDPs. Instead, for a basic RL system, you can consider modifying hyperparameters or focusing on key events as suggested above. Outside of a basic RL system there could be merit in replicating other things, such as "instinctive" fear.
  
  > **Comments:**
  > **username_2**: @DuttaA: I agree, which is why I suggest a bunch of stuff that is not "real fear (tm)". I think a very basic "instinctive fear" using RL would involve somehow adding a prior low value into the value function programmatically without actual experience.  
  
  ---
  
  ### Answer by username_3 on 2018-10-05. Score: 2
  
  Fear of this kind is an irrational response (large negative incentive in response to a small risk). Modeling fear would need to model a "grossness" factor associated with, for example, spiders so that the normally un-proportional response would occur. The "grossness" factor could be manifested in many other forms to magnify a response to a previously unpleasant, though not particularly dangerous, experience. Such fear can also be inspired by hearsay (think hysteria caused by a sensational news story).
  A NN would normally only respond minimally to a minimal risk.
  
  
  ---
  
  ### Answer by username_4 on 2018-08-12. Score: 2
  
  I think there are 2 ways to make this happen: 1) explicitly program fear as a constraint or parameter in some logical expression, or 2) utilize a large set of training data to teach fear.
  
  Think about a basic Pacman game-- whether Pacman fears the ghosts or doesn't fear them is hard to tell, but they ARE ghosts and Pacman avoids them so I think it's safe we can use this as a basic example of "fear". Since, in this game, fear = avoidance, you could logically program avoidance to be some sort of distance. I tried this with Pacman reinforcement learning. I tried to set a distance of 5 squares to the ghosts and anytime Pacman could see a ghost within 5 squares, he would move in a different direction. What I found is that while Pacman will try to avoid ghosts, he doesn't know strategy (or have intelligence). Pacman would simply move away from ghosts until he got boxed in.
  
  My point is that you can program your network to avoid spiders to not get bit, but without training, you will just be creating a basic parameter that might cause problems if there are 100 super aggressive spiders coming at you! The better way is to use some base logic to avoid spiders, but then train the network to be rewarded the better spiders are avoided.
  
  Now, there are many situations of fear so this one example with Pacman would not necessarily apply to all... Just trying to give some insight in my experience with teaching fear with reinforcement learning in Pacman.
  
  > **Comments:**
  > **username_0**: I'm thinking that for some things like fire it needs to be an instinct. Because by the time you've had a few "training examples" you'd be pretty burned. But with other things you should only need one example like getting bit by a hamster you should learn hamsters have sharp teeth so don't put your hand in their mouth. So for some things instincts should just prevent you from doing things like putting your hand in a fire. Or make you jump back. But should you also be scared of going near fire if you learn that you might get hit by a spark?  
  > **username_5**: @zooby in my experience little kids don't really learn to fear the hot stove or fire until they do get burned.  (s'why you've got to keep such a close eye on them!)  I'd say that if the algorithm is learning avoidance through a technique like reinforcement learning, it is "learning what to fear", whereas, if it has the avoidance mechanism pre-programmed, that would be "instinct".  
  
  ---
  
  ### Answer by username_6 on 2018-08-14. Score: 1
  
  I would suggest having the agent weight its learning from a given event based on the severity of the consequences for that event happening. Eg. Have it develop a threat model like those typically drafted up in the Information Security field. High risk but low probability is something that can be accounted for and judged against.
  
  Trying to directly imitate human fear would be silly, you'd likely end up with AIs that have phobias if you succeeded too well.
  
  
  ---
  
  