Answers.Body: <p>For the actor, I'd say the 'not mutually exclusive' option is more biologically plausible in the context of muscle systems, where the actions can be seen as simultaneous muscle activations. Maybe at a higher level, an agent thinks of the action as 'go north' or 'go south', but the final outputs which have to control muscles at a lower level have to represent simultaneous muscle activations.</p>
  
  <p>For the critic, I'd say the 'single output node' is more biologically plausible. Agents perceive the world in the form of high dimensional inputs, such as images. The approach where a value function is learned in a tabular fashion where you know the value for every single state doesn't really scale very well and is limited to small discrete state spaces. For biological agents, it makes sense to have a function that senses the current state of the environment and outputs a single number that represents the value, which gives the agent an idea of how things are going so far given the actions it took in the past.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T01:17:30.343Z
Answers.Text: Hi @yewang,  
  
  Yes I agree, in fact the mutually exclusive actor (1) will also not scale well in my opinion.      
  
  -Do you know of any models with plausible (eg local) learning rules that support the plausible representations (2&3) ?    
  
  
  thanks
Answers.UserDisplayName: thward
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T05:35:53.643Z
Answers.Text: For example, if you use A2C to control a robot, a common approach is to train a neural network that outputs a probability distribution over a number of actuators, often as a multivariate gaussian distribution, where the actor outputs the mean of the multivariate distribution (compatible with (2)). For the value function, it's common to model it as a function that takes an n-dimensional vector (for example joint angles) and outputs a single number representing the value of the state (compatible with (3)). Since the state space is continuous, you can't store a value for every state in a table.
Answers.UserDisplayName: yewang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-17T06:03:25.863Z
Answers.Text: Hi @yewang, thanks. The examples I looked at for A2C I think are using back-propagation to train the value function. -  Do you know of a paper that achieves this using only local rules ?
Answers.UserDisplayName: thward
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-17T09:13:52.830Z
Answers.Text: what do you mean by local rules? I think backpropagation can be thought of as local rules if you consider that the update rule for a neuron depends only on immediately close neurons. As you propagate the gradients layer after layer you get a global behavior in an indirect way, but the basic rules are local.
Answers.UserDisplayName: yewang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-19T08:39:31.713Z
Answers.Text: Hi @yewang, I appreciate your feedback :) By local rules I meant it only uses local information available to each synapse. Principally hebbian type rules, including 3-factor rules (pre, post & neuromodulator). So in the case of A2C I believe the actor could be implemented with a local 3-factor rule (eg a Policy Gradient method such as ARP). But the critic value function appears to use backprop. I have not found a hebbian type rule to support that single node representation.FYI However I happen to have devised a simple method to do this, I can give you the details if you're interested.
Answers.UserDisplayName: thward
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T08:32:37.753Z
Answers.LastActivityDate: 2019-02-15T08:32:37.753Z
Answers.UserDisplayName: yewang
Body: <p>Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either</p>
  
  <ol>
  <li><p>mutually exclusive with each other (e.g., go north, go south, etc),
  achieved by winner-takes-all. </p></li>
  <li><p>NOT mutually exclusive with each other (e.g. left leg forward, right leg forward); these actions may occur concurrently. To go north, the correct combination of nodes must be active.</p></li>
  </ol>
  
  <p>Similarly which representation is most plausible for critic output nodes?</p>
  
  <ol start="3">
  <li><p>A single output node that outputs a real number representing the
  reward. </p></li>
  <li><p>A set of output nodes each representing a separate value, achieved by winner-takes-all.</p></li>
  </ol>
  
  <p>Or do other representations better align with real brains ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-14T11:43:59.887Z
Comments.Text: To make it clear, you want to represent both the actor and the critic, in the actor-critic architecture, as a neural network. However, it is not clear to me what your actual problem is. I understand you want to be consistent with the biology (if possible), but what exactly must be consistent with the biology, in this context? What nodes of the last layer of the neural network (both for the actor and the critic)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-16T03:29:44.340Z
Comments.Text: Hi @nbro,
  
  To be biologically plausible the model should really apply to all problems, rather than a specific problem. 
  
  Some interesting example..
  
  [This](https://papers.nips.cc/paper/2749-brain-inspired-reinforcement-learning.pdf) model attempts to tie in with brain regions, however it uses a winner-takes-all rule for the action.  
  
  [This](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003024) model uses plausible local learning rules for neurons, but uses a winner-takes-all rule for the action and I think requires a separate critic neuron for each state.
Comments.UserDisplayName: thward
Tags: <neural-networks><reinforcement-learning><actor-critic-methods>
Title: What is the most biologically plausible representation for the actor and critic?
UserDisplayName: thward
ThreadText: # What is the most biologically plausible representation for the actor and critic?
  
  (Asked by: username_0 on 2019-02-14)
  
  Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either
  
  1. mutually exclusive with each other (e.g., go north, go south, etc),
     achieved by winner-takes-all.
  2. NOT mutually exclusive with each other (e.g. left leg forward, right leg forward); these actions may occur concurrently. To go north, the correct combination of nodes must be active.
  
  Similarly which representation is most plausible for critic output nodes?
  
  3. A single output node that outputs a real number representing the
     reward.
  4. A set of output nodes each representing a separate value, achieved by winner-takes-all.
  
  Or do other representations better align with real brains ?
  
  **Tags:** neural-networks, reinforcement-learning, actor-critic-methods
  
  > **Comments:**
  > **username_1**: To make it clear, you want to represent both the actor and the critic, in the actor-critic architecture, as a neural network. However, it is not clear to me what your actual problem is. I understand you want to be consistent with the biology (if possible), but what exactly must be consistent with the biology, in this context? What nodes of the last layer of the neural network (both for the actor and the critic)?  
  > **username_0**: Hi @username_1,  To be biologically plausible the model should really apply to all problems, rather than a specific problem.   Some interesting example..  [This](https://papers.nips.cc/paper/2749-brain-inspired-reinforcement-learning.pdf) model attempts to tie in with brain regions, however it uses a winner-takes-all rule for the action.    [This](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003024) model uses plausible local learning rules for neurons, but uses a winner-takes-all rule for the action and I think requires a separate critic neuron for each state.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-02-15. Score: 1
  
  For the actor, I'd say the 'not mutually exclusive' option is more biologically plausible in the context of muscle systems, where the actions can be seen as simultaneous muscle activations. Maybe at a higher level, an agent thinks of the action as 'go north' or 'go south', but the final outputs which have to control muscles at a lower level have to represent simultaneous muscle activations.
  
  For the critic, I'd say the 'single output node' is more biologically plausible. Agents perceive the world in the form of high dimensional inputs, such as images. The approach where a value function is learned in a tabular fashion where you know the value for every single state doesn't really scale very well and is limited to small discrete state spaces. For biological agents, it makes sense to have a function that senses the current state of the environment and outputs a single number that represents the value, which gives the agent an idea of how things are going so far given the actions it took in the past.
  
  > **Comments:**
  > **username_0**: Hi @username_2,    Yes I agree, in fact the mutually exclusive actor (1) will also not scale well in my opinion.        -Do you know of any models with plausible (eg local) learning rules that support the plausible representations (2&3) ?       thanks  
  > **username_2**: For example, if you use A2C to control a robot, a common approach is to train a neural network that outputs a probability distribution over a number of actuators, often as a multivariate gaussian distribution, where the actor outputs the mean of the multivariate distribution (compatible with (2)). For the value function, it's common to model it as a function that takes an n-dimensional vector (for example joint angles) and outputs a single number representing the value of the state (compatible with (3)). Since the state space is continuous, you can't store a value for every state in a table.  
  > **username_0**: Hi @username_2, thanks. The examples I looked at for A2C I think are using back-propagation to train the value function. -  Do you know of a paper that achieves this using only local rules ?  
  > **username_2**: what do you mean by local rules? I think backpropagation can be thought of as local rules if you consider that the update rule for a neuron depends only on immediately close neurons. As you propagate the gradients layer after layer you get a global behavior in an indirect way, but the basic rules are local.  
  > **username_0**: Hi @username_2, I appreciate your feedback :) By local rules I meant it only uses local information available to each synapse. Principally hebbian type rules, including 3-factor rules (pre, post & neuromodulator). So in the case of A2C I believe the actor could be implemented with a local 3-factor rule (eg a Policy Gradient method such as ARP). But the critic value function appears to use backprop. I have not found a hebbian type rule to support that single node representation.FYI However I happen to have devised a simple method to do this, I can give you the details if you're interested.  
  
  ---
  
  