<title>Introduction</title> <p>Numerous experimental studies (see ##REF##11127835##[1]## for a review; ##REF##17287502##[2]## discusses more recent
in-vivo results) have shown that the efficacy of synapses changes in dependence of
the time difference
Δ<italic>t</italic> = <italic>t<sub>post</sub></italic>−<italic>t<sub>pre</sub></italic>
between the firing times <italic>t<sub>pre</sub></italic> and
<italic>t<sub>post</sub></italic> of the pre- and postsynaptic neurons. This
effect is called spike-timing-dependent plasticity (STDP). But a major puzzle for
understanding learning in biological organisms is the relationship between
experimentally well-established rules for STDP on the microscopic level, and
adaptive changes of the behavior of biological organisms on the macroscopic level.
Neuromodulatory systems, which send diffuse signals related to reinforcements
(rewards) and behavioral state to several large networks of neurons in the brain,
have been identified as likely intermediaries that relate these two levels of
plasticity. It is well-known that the consolidation of changes of synaptic weights
in response to pre- and postsynaptic neuronal activity requires the presence of such
third signals ##REF##11252764##[3]##,##REF##12031406##[4]##. In particular, it has been demonstrated that
dopamine (which is behaviorally related to novelty and reward prediction ##REF##17400301##[5]##) gates
plasticity at corticostriatal synapses ##REF##11544526##[6]##,##REF##12371508##[7]## and within the cortex
##REF##11452310##[8]##. It
has also been shown that acetylcholine gates synaptic plasticity in the cortex (see
for example ##REF##10676963##[9]## and ##REF##12165477##[10]##,##REF##15242655##[11]## contains a nice review of the literature).</p> <p>Corresponding spike-based rules for synaptic plasticity of the formhave been proposed in ##REF##17220510##[12]## and ##UREF##0##[13]## (see
##FIG##0##Figure 1## for an illustration
of this learning rule), where <italic>w<sub>ji</sub></italic> is the weight of a
synapse from neuron <italic>i</italic> to neuron <italic>j</italic>,
<italic>c<sub>ji</sub></italic>(<italic>t</italic>) is an eligibility trace
of this synapse which collects weight changes proposed by STDP, and
<italic>d</italic>(<italic>t</italic>) = <italic>h</italic>(<italic>t</italic>)−<italic>h̅</italic>
results from a neuromodulatory signal <italic>h</italic>(<italic>t</italic>) with
mean value <italic>h̅</italic>. It was shown in ##REF##17220510##[12]## that a number of
interesting learning tasks in large networks of neurons can be accomplished with
this simple rule in Equation 1. It has recently been shown that quite similar
learning rules for spiking neurons arise when one applies the general framework of
distributed reinforcement learning from ##UREF##1##[14]## to networks of spiking
neurons ##UREF##0##[13]##,##REF##17571943##[15]##, or if one maximizes the likelihood of
postsynaptic firing at desired firing times ##REF##16764506##[16]##. However no analytical
tools have been available, which make it possible to predict for what learning
tasks, and under which parameter settings, reward-modulated STDP will be successful.
This article provides such analytical tools, and demonstrates their applicability
and significance through a variety of computer simulations. In particular, we
identify conditions under which neurons can learn through reward-modulated STDP to
classify temporal presynaptic firing patterns, and to respond with particular spike
patterns.</p> <p>We also provide a model for the remarkable operant conditioning experiments of ##REF##4196269##[17]## (see also
##REF##4974291##[18]##,##REF##17234689##[19]##). In the simpler ones of these experiments the
spiking activity of single neurons (in area 4 of the precentral gyrus of monkey
cortex) was recorded, the deviation of the current firing rate of an arbitrarily
selected neuron from its average firing rate was made visible to the monkey through
the displacement of an illuminated meter arm, whose rightward position corresponded
to the threshold for the feeder discharge. The monkey received food rewards for
increasing (or in alternating trials for decreasing) the firing rate of this neuron.
The monkeys learnt quite reliably (within a few minutes) to change the firing rate
of this neuron in the currently rewarded direction. Adjacent neurons tended to
change their firing rate in the same direction, but also differential changes of
directions of firing rates of pairs of neurons are reported in ##REF##4196269##[17]## (when these differential
changes were rewarded). For example, it was shown in Figure 9 of ##REF##4196269##[17]## (see also Figure 1 in ##REF##17234689##[19]##) that pairs of neurons
that were separated by no more than a few hundred microns could be independently
trained to increase or decrease their firing rates. Obviously the existence of
learning mechanisms in the brain which are able to solve this extremely difficult
credit assignment problem provides an important clue for understanding the
organization of learning in the brain. We examine in this article analytically under
what conditions reward-modulated STDP is able to solve such learning problem. We
test the correctness of analytically derived predictions through computer
simulations of biologically quite realistic recurrently connected networks of
neurons, where an increase of the firing rate of one arbitrarily selected neuron
within a network of 4000 neurons is reinforced through rewards (which are sent to
all 142813 synapses between excitatory neurons in this recurrent network). We also
provide a model for the more complex operant conditioning experiments of ##REF##4196269##[17]## by
showing that pairs of neurons can be differentially trained through reward-modulated
STDP, where one neuron is rewarded for increasing its firing rate, and
simultaneously another neuron is rewarded for decreasing its firing rate. More
precisely, we increased the reward signal <italic>d</italic>(<italic>t</italic>)
which is transmitted to all synapses between excitatory neurons in the network
whenever the first neuron fired, and decreased this reward signal whenever the
second neuron fired (the resulting composed reward corresponds to the displacement
of the meter arm that was shown to the monkey in these more complex operant
conditioning experiments).</p> <p>Our theory and computer simulations also show that reward-modulated STDP can be
applied to all synapses within a large network of neurons for long time periods,
without endangering the stability of the network. In particular this synaptic
plasticity rule keeps the network within the asynchronous irregular firing regime,
which had been described in ##REF##11165912##[20]## as a dynamic regime that resembles spontaneous
activity in the cortex. Another interesting aspect of learning with reward-modulated
STDP is that it requires spontaneous firing and trial-to-trial variability within
the networks of neurons where learning takes place. Hence our learning theory for
this synaptic plasticity rule provides a foundation for a functional explanation of
these characteristic features of cortical network of neurons that are undesirable
from the perspective of most computational theories.</p>
<title>Methods</title> <p>We first describe the simple neuron model that we used for the theoretical analysis,
and then provide derivations of the equations that were discussed in the preceding
section. After that we describe the models for neurons, synapses, and synaptic
background activity (“noise”) that we used in the computer
simulations. Finally we provide technical details to each of the 5 computer
simulations that we discussed in the preceding section.</p> <title>Linear Poisson Neuron Model</title> <p>In our theoretical analysis, we use a linear Poisson neuron model whose output
spike train  is a realization of a Poisson process with the underlying
instantaneous firing rate <italic>R<sub>j</sub></italic>(<italic>t</italic>).
The effect of a spike of presynaptic neuron <italic>i</italic> at time
<italic>t</italic>′ on the membrane potential of neuron
<italic>j</italic> is modeled by an increase in the instantaneous firing rate by
an amount
<italic>w<sub>ji</sub></italic>(<italic>t</italic>′)<italic>ε</italic>(<italic>t</italic>−<italic>t</italic>′),
where <italic>ε</italic> is a response kernel which models the time
course of a postsynaptic potential (PSP) elicited by an input spike. Since STDP
according to ##REF##17220510##[12]## has been experimentally confirmed only for
excitatory synapses, we will consider plasticity only for excitatory connections
and assume that <italic>w<sub>ji</sub></italic>≥0 for all
<italic>i</italic> and <italic>ε</italic>(<italic>s</italic>)≥0
for all <italic>s</italic>. Because the synaptic response is scaled by the
synaptic weights, we can assume without loss of generality that the response
kernel is normalized to . In this linear model, the contributions of all inputs are
summed up linearly:where
<italic>S</italic>
<sub>1</sub>,…,<italic>S<sub>n</sub></italic> are
the <italic>n</italic> presynaptic spike trains. Since the instantaneous firing
rate <italic>R</italic>(<italic>t</italic>) is analogous to the membrane
potential of other neuron models, we occasionally refer to
<italic>R</italic>(<italic>t</italic>) as the “membrane
potential” of the neuron.</p> <title>Learning Equations</title> <p>In the following, we denote by  the ensemble average of a random variable <italic>x</italic>
given that neuron <italic>k</italic> spikes at time <italic>t</italic> and
neuron <italic>i</italic> spikes at time <italic>t</italic>′. We will
also sometimes indicate the variables
<italic>Y</italic>
<sub>1</sub>,<italic>Y</italic>
<sub>2</sub>,…
over which the average of <italic>x</italic> is taken by writing .</p> <title>Derivation of Equation 6</title> <p>Using Equations 5, 1, and 4, we obtain the expected weight change between
time <italic>t</italic> and <italic>t</italic>+<italic>T</italic>
with
<italic>D<sub>ji</sub></italic>(<italic>t</italic>,<italic>s</italic>,<italic>r</italic>) = 〈<italic>d</italic>(<italic>t</italic>)|Neuron
<italic>j</italic> spikes at
<italic>t</italic>−<italic>s</italic>, and neuron
<italic>i</italic> spikes at
<italic>t</italic>−<italic>s</italic>−<italic>r</italic>〉<italic><sub>E</sub></italic>, and the joint firing rate
<italic>ν<sub>ji</sub></italic>(<italic>t</italic>,<italic>r</italic>) = 〈<italic>S<sub>j</sub></italic>(<italic>t</italic>)<italic>S<sub>i</sub></italic>(<italic>t</italic>−<italic>r</italic>)〉<italic><sub>E</sub></italic> describes correlations between spike timings of neurons
<italic>j</italic> and <italic>i</italic>. The joint firing rate
<italic>ν<sub>ji</sub></italic>(<italic>t</italic>−<italic>s</italic>,<italic>r</italic>)
depends on the weight at time
<italic>t</italic>−<italic>s</italic>. If the learning rate
defined by the magnitude of <italic>W</italic>(<italic>r</italic>) is small,
the synaptic weights can be assumed constant on the time scale of
<italic>T</italic>. Thus, the time scales of neuronal dynamics are separated
from the slow time scale of learning. For slow learning, synaptic weights
integrate a large number of small changes. We can then expect that averaged
quantities enter the learning dynamics. In this case, we can argue that
fluctuations of a weight <italic>w<sub>ji</sub></italic> about its mean are
negligible and it can well be approximated by its average
〈<italic>w<sub>ji</sub></italic>〉<italic><sub>E</sub></italic> (it is “self-averaging”, see ##UREF##2##[21]##,##UREF##5##[36]##). To ensure
that average quantities enter the learning dynamics, many presynaptic and
postsynaptic spikes as well as many independently delivered rewards at
varying delays have to occur within <italic>T</italic>. Hence, in general,
the time scale of single spike occurrences and the time scale of the
eligibility trace is required to be much smaller than the time scale of
learning. If time scales can be separated, we can drop the expectation on
the left hand side of the last equation and writeWe thus obtain Equation 6:
</p> <title>Simplification of Equation 6</title> <p>In order to simplify this equation, we first observe that
<italic>W</italic>(<italic>r</italic>) is vanishing for large
|<italic>r</italic>|. Hence we can approximate the integral over the
learning window by a bounded integral  for some <italic>T<sub>W</sub></italic>&gt;0 and
<italic>T<sub>W</sub></italic>≪<italic>T</italic>. In the
analyzes of this article, we consider the case where reward is delivered
with a relatively large temporal delay. To be more precise, we assume that a
pre-post spike pair has an effect on the reward signal only after some
minimal delay <italic>d<sub>r</sub></italic> and that we can write  for some baseline reward <italic>d</italic>
<sub>0</sub>
and a part which depends on the timing of pre-post spike pairs with  for
<italic>s</italic>&lt;<italic>d<sub>r</sub></italic> and
<italic>d<sub>r</sub></italic>&gt;<italic>T<sub>W</sub></italic>. We
can then approximate the second term of Equation 6:because
〈<italic>ν<sub>ji</sub></italic>(<italic>t</italic>−<italic>s</italic>−<italic>r</italic>,<italic>r</italic>)〉<italic><sub>T</sub></italic>≈〈<italic>ν<sub>ji</sub></italic>(<italic>t</italic>−<italic>s</italic>,<italic>r</italic>)〉<italic><sub>T</sub></italic> for
<italic>r</italic>∈[−<italic>T<sub>W</sub></italic>,<italic>T<sub>W</sub></italic>]
and <italic>T<sub>W</sub></italic>≪<italic>T</italic>. Since  for
<italic>s</italic>≤<italic>T<sub>W</sub></italic>, the second term in
the brackets is equivalent to  which in turn is approximately given by  if we assume that
<italic>f<sub>c</sub></italic>(<italic>s</italic>+<italic>r</italic>)≈<italic>f<sub>c</sub></italic>(<italic>s</italic>)
for <italic>s</italic>≥<italic>d<sub>r</sub></italic> and
|<italic>r</italic>|&lt;<italic>T<sub>W</sub></italic>. We can
thus approximate the second term of Equation 6 asWith this approximation, the first and second term of
Equation 6 can be combined in a single integral to obtain Equation 8.</p> <title>Derivations for the Biofeedback Experiment</title> <p>We assume that a reward with the functional form
<italic>ε<sub>r</sub></italic> is delivered for each postsynaptic spike
with a delay <italic>d<sub>r</sub></italic>. The reward as time
<italic>t</italic> is therefore
</p> <title>Weight change for the reinforced neuron (derivation of Equation 10)</title> <p>The reward correlation for a synapse <italic>ki</italic> afferent to the
reinforced neuron is If we assume that the output firing rate is constant on the time
scale of the reward function, the first term vanishes. We rewrite the result asThe mean weight change for weights to the reinforced neuron is thereforeWe show that the second term in the brackets is very small
compared to the first term:The last approximation is based on the assumption that
<italic>f<sub>c</sub></italic>(<italic>s</italic>)≈<italic>f<sub>c</sub></italic>(<italic>s</italic>−<italic>r</italic>′)
and
〈<italic>ν<sub>ki</sub></italic>(<italic>t</italic>−<italic>r</italic>′,<italic>r</italic>)〉<italic><sub>T</sub></italic>≈〈<italic>ν<sub>ki</sub></italic>(<italic>t</italic>,<italic>r</italic>)〉<italic><sub>T</sub></italic> for
<italic>r</italic>′∈[−<italic>T<sub>W</sub></italic>−<italic>T<sub>ε</sub></italic>,<italic>T<sub>W</sub></italic>].
Here, <italic>T<sub>W</sub></italic> is the time scale of the learning window
(see above), and <italic>T<sub>ε</sub></italic> is time scale of the
PSP, i.e., we have <italic>ε</italic>(<italic>s</italic>)≈0 for
<italic>s</italic>≥<italic>T<sub>ε</sub></italic>. Since  by definition, we see that this is the first term in the
brackets of Equation 20 scaled by <italic>w<sub>ki</sub></italic>. For neurons
with many input synapses we have <italic>w<sub>ki</sub></italic>≪1. Thus
the second term in the brackets of Equation 20 is small compared to the first
term. We therefore have
</p> <title>Weight change for non-reinforced neurons (derivation of Equation 11)</title> <p>The reward correlation of a synapse <italic>ji</italic> to a non-reinforced
neuron <italic>j</italic> is given byWe havefor which we obtainIn analogy to the previous derivation, we assume here that the
firing rate
<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>)
in the denominator results from many PSPs. Hence, the single PSP
<italic>w<sub>ji</sub><italic>ε</italic></italic>(<italic>r</italic>)
is small compared to
<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>).
Similarly, we assume that with weights <italic>w<sub>ki</sub></italic>,
<italic>w<sub>ji</sub></italic>≪1, the second term in the
nominator is small compared to the joint firing rate
<italic>ν<sub>kj</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′,<italic>s</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′).
We therefore approximate the reward correlation byHence, the reward correlation of a non-reinforced neuron depends
on the correlation of this neuron with the reinforced neuron. The mean weight
change for a non-reinforced neuron <italic>j</italic>≠<italic>k</italic>
is thereforeThis equation deserves a remark for the case that
<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>)
is zero, since it appears in the denominator of the fraction. Note that in this
case, both
<italic>ν<sub>kj</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′,<italic>s</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′)
and
<italic>ν<sub>ji</sub></italic>(<italic>t</italic>−<italic>s</italic>,<italic>r</italic>)
are zero. In fact, if we take the limit
<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>)→0,
then both of these factors approach zero at least as fast. Hence, in the limit
of
<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>)→0,
the term in the angular brackets evaluates to zero. This reflects the fact that
since STDP is driven by pre- and postsynaptic spikes, there is no weight change
if no postsynaptic spikes occur.</p> <title>For uncorrelated neurons, Equation 11 evaluates to zero</title> <p>For uncorrelated neurons <italic>k</italic>, <italic>j</italic>,
<italic>ν<sub>kj</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′,<italic>s</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′)
can be factorized into
<italic>ν<sub>k</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′)<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>),
and we obtainThis evaluates approximately to zero if the mean output rate
of neuron <italic>k</italic> is constant on the time scale of the reward
kernel.</p> <title>Analysis of Spike-Timing-Dependent Rewards (Derivation of Conditions
13–15)</title> <p>Below, we will indicate the variables
<italic>Y</italic>
<sub>1</sub>,<italic>Y</italic>
<sub>2</sub>,…
over which the average of <italic>x</italic> is taken by writing . From Equation 12, we can determine the reward correlation for
synapse <italic>i</italic>
where  denotes the instantaneous firing rate of the trained neuron at
time <italic>t</italic>, and
<italic>ν</italic>
<sup>*</sup>(<italic>t</italic>) = 〈<italic>S</italic>
<sup>*</sup>(<italic>t</italic>)〉<italic><sub>E</sub></italic> denotes the instantaneous rate of the target spike train at time
<italic>t</italic>. Since weights are changing very slowly, we have
<italic>w<sub>ji</sub></italic>(<italic>t</italic>−<italic>s</italic>−<italic>r</italic>)≈<italic>w<sub>ji</sub></italic>(<italic>t</italic>).
In the following, we will drop the dependence of <italic>w<sub>ji</sub></italic>
on <italic>t</italic> for brevity. For simplicity, we assume that input rates
are stationary and uncorrelated. In this case (since the weights are changing
slowly), also the correlations between inputs and outputs can be assumed
stationary,
<italic>ν<sub>ji</sub></italic>(<italic>t</italic>,<italic>r</italic>) = <italic>ν<sub>ji</sub></italic>(<italic>r</italic>).
With constant input rates, we can rewrite Equation 21 aswith . We use this results to obtain the temporally smoothed weight
change for synapse <italic>ji</italic>. With stationary correlations, we can
drop the dependence of <italic>ν<sub>ji</sub></italic> on
<italic>t</italic> and write
<italic>ν<sub>ji</sub></italic>(<italic>t</italic>,<italic>r</italic>) = <italic>ν<sub>ji</sub></italic>(<italic>r</italic>).
Furthermore, we define  and obtainWe assume that the eligibility function
<italic>f<sub>c</sub></italic>(<italic>d<sub>r</sub></italic>)≈<italic>f<sub>c</sub></italic>(<italic>d<sub>r</sub></italic>+<italic>r</italic>)
if |<italic>r</italic>| is on the time scale of a PSP, the learning window, or
the reward kernel, and that <italic>d<sub>r</sub></italic> is large compared to
these time scales. Then, we havewhere  is the convolution of the reward kernel with the PSP.
Furthermore, we findWith these simplifications, and the abbreviation  we obtain the weight change at synapse <italic>ji</italic>
where .</p> <p>For uncorrelated Poisson input spike trains of rate  and the linear Poisson neuron model, the input-output
correlations are . With these correlations, we obtain  where , and. The weight change at synapse <italic>ji</italic> is then
</p> <p>We will now bound the expected weight change for synapses <italic>ji</italic>
with  and for synapses <italic>jk</italic> with . In this way we can derive conditions for which the expected
weight change for the former synapses is positive, and that for the latter type
is negative. First, we assume that the integral over the reward kernel is
positive. In this case, the weight change given by Equation 22 is negative for
synapses <italic>i</italic> with  if and only if , and . In the worst case, <italic>w<sub>ji</sub></italic> is
<italic>w<sub>max</sub></italic> and  is small. We have to guarantee some minimal output rate  such that even if
<italic>w<sub>ji</sub></italic> = <italic>w<sub>max</sub></italic>,
this inequality is fulfilled. This could be guaranteed by some noise current.
Given such minimal output rate, we can state the first inequality which
guarantees convergence of weights <italic>w<sub>ji</sub></italic> with 
For synapses <italic>ji</italic> with , we obtain two more conditions. The approximate weight change
is given byThe last term in this equation is positive and small. We can
ignore it in our sufficient condition. The second to last term is negative. We
will include in our condition that the third to last term compensates for this
negative term. Hence, the second condition iswhich should be satisfied in most setups. If we assume that this
holds, we obtainwhich should be positive. We obtain the following inequalityAll three inequalities are summarized in the following:where  is the maximal output rate. If these inequalities are
fulfilled and input rates are positive, then the weight vector converges on
average from any initial weight vector to <bold>w</bold>*. The second
condition is less severe, and should be easily fulfilled in most setups. If this
is the case, the first Condition 13 ensures that weights with
<italic>w</italic>* = 0 are depressed
while the third Condition 15 ensures that weights with
<italic>w</italic>* = <italic>w<sub>max</sub></italic>
are potentiated.</p> <title>Analysis of the Pattern Discrimination Task (Derivation of Equation 17)</title> <p>We assume that a trial consists of the presentation of a single pattern starting
at time <italic>t</italic> = 0. We compute the
weight change for a single trial given that pattern
<italic>X</italic>∈{<italic>P</italic>,<italic>N</italic>} was
presented with the help of Equations 1, 3, and 4 asWe can compute the average weight change given that pattern
<italic>X</italic> was presented:If we assume that <italic>f<sub>c</sub></italic> is approximately
constant on the time scale of the learning window <italic>W</italic>, we can
simplify this toFor the linear Poisson neuron, we can write the auto-correlation
function aswhere
<italic>ν<sup>X</sup></italic>(<italic>t</italic>) = 〈<italic>S<sup>post</sup></italic>(<italic>t</italic>)〉<italic><sub>E</sub></italic>
<sub>|<italic>X</italic></sub> is the ensemble average rate at time
<italic>t</italic> given that pattern <italic>X</italic> was presented. If
an experiment for a single pattern runs over the time interval
[0,<italic>T</italic>′], we can compute the
total average weight change  of a trial given that pattern <italic>X</italic> was presented asBy definingwe can write Equation 23 asWe assume that eligibility traces and reward signals have settled
to zero before a new pattern is presented. The expected weight change for the
successive presentation of both patterns is thereforeThe equations can easily be generalized to the case where
multiple input spikes per synapse are allowed and where jitter on the templates
is allowed. However, the main effect of the rule can be read off the equations
given here.</p> <title>Common Models and Parameters of the Computer Simulations</title> <p>We describe here the models and parameter values that were used in all our
computer simulations. We will specify in a subsequent section the values of
other parameters that had to be chosen differently in individual computer
simulations, in dependence of their different setups and requirements of each
computer simulation.</p> <title>LIF Neuron Model</title> <p>For the computer simulations LIF neurons with conductance-based synapses were
used. The membrane potential <italic>V<sub>m</sub></italic>(<italic>t</italic>)
of this neuron model is given by:where <italic>C<sub>m</sub></italic> is the membrane capacitance,
<italic>R<sub>m</sub></italic> is the membrane resistance,
<italic>V<sub>resting</sub></italic> is the resting potential, and
<italic>g<sub>e</sub></italic>
<sub>,<italic>j</italic></sub>(<italic>t</italic>)
and
<italic>g<sub>i</sub></italic>
<sub>,<italic>j</italic></sub>(<italic>t</italic>)
are the <italic>K<sub>e</sub></italic> and <italic>K<sub>i</sub></italic>
synaptic conductances from the excitatory and inhibitory synapses respectively.
The constants <italic>E<sub>e</sub></italic> and <italic>E<sub>i</sub></italic>
are the reversal potentials of excitatory and inhibitory synapses.
<italic>I<sub>noise</sub></italic> represents the synaptic background
current which the neuron receives (see below for details).</p> <p>Whenever the membrane potential reaches a threshold value
<italic>V<sub>thresh</sub></italic>, the neuron produces a spike, and its
membrane potential is reset to the value of the reset potential
<italic>V<sub>reset</sub></italic>. After a spike, there is a refractory
period of length <italic>T<sub>refract</sub></italic>, during which the membrane
potential of the neuron remains equal to the value
<italic>V<sub>m</sub></italic>(<italic>t</italic>) = <italic>V<sub>reset</sub></italic>.
After the refractory period <italic>V<sub>m</sub></italic>(<italic>t</italic>)
continues to change according to Equation 24.</p> <p>For a given synapse, the dynamics of the synaptic conductance
<italic>g</italic>(<italic>t</italic>) is defined bywhere <italic>A</italic>(<italic>t</italic>) is the amplitude of
the postsynaptic response (PSR) to a single presynaptic spike, which varies over
time due to the inherent short-term dynamics of the synapse, and
{<italic>t</italic>
<sup>(<italic>k</italic>)</sup>} are the spike times
of the presynaptic neuron. The conductance of the synapse decreases
exponentially with time constant <italic>τ<sub>syn</sub></italic>, and
increases instantaneously by amount of <italic>A</italic>(<italic>t</italic>)
whenever the presynaptic neuron spikes.</p> <p>In all computer simulations we used the following values for the neuron and
synapse parameters. The membrane resistance of the neurons was
<italic>R<sub>m</sub></italic> = 100
MΩ, the membrane capacitance
<italic>C<sub>m</sub></italic> = 0.3 nF, the
resting potential, reset potential and the initial value of the membrane
potential had the same value of
<italic>V<sub>resting</sub></italic> = <italic>V<sub>reset</sub></italic> = <italic>V<sub>m</sub></italic>(0) = −70
mV, the threshold potential was set to
<italic>V<sub>thresh</sub></italic> = −59
mV and the refractory period
<italic>T<sub>refract</sub></italic> = 5 ms.
For the synapses we used a time constant set to
<italic>τ<sub>syn</sub></italic> = 5
ms, reversal potential
<italic>E<sub>e</sub></italic> = 0 mV for the
excitatory synapses and
<italic>E<sub>e</sub></italic> = −75
mV for the inhibitory synapses. All synapses had a synaptic delay of
<italic>t<sub>delay</sub></italic> = 1
ms.</p> <title>Short-Term Dynamics of Synapses</title> <p>We modeled the short-term dynamics of synapses according to the phenomenological
model proposed in ##REF##9560274##[37]##, where the amplitude
<italic>A<sub>k</sub></italic> = <italic>A</italic>(<italic>t<sub>k</sub></italic>+<italic>t<sub>delay</sub></italic>)
of the postsynaptic response for the <italic>k</italic>th spike in a spike train
with inter-spike intervals Δ<sub>1</sub>,Δ<sub>2</sub>,…,Δ<italic><sub>k</sub></italic>
<sub>−1</sub> is calculated with the following equationswith hidden dynamic variables
<italic>u</italic>∈[0,1] and
<italic>R</italic>∈[0,1] whose initial values for the
1st spike are
<italic>u</italic>
<sub>1</sub> = <italic>U</italic>
and <italic>R</italic> = 1 (see ##REF##12022505##[38]## for
a justification of this version of the equations, which corrects a small error
in ##REF##9560274##[37]## ). The variable <italic>w</italic> is the
synaptic weight which scales the amplitudes of postsynaptic responses. If
long-term plasticity is introduced, this variable is a function of time. In the
simulations, for the neurons in the circuits the values for the U, D and F
parameters were drawn from Gaussian distributions with mean values which
depended on whether the type of presynaptic and postsynaptic neuron of the
synapse is excitatory or inhibitory, and were chosen according to the data
reported in ##REF##9560274##[37]## and ##REF##10634775##[39]##. The mean values of
the Gaussian distributions are given in ##TAB##1##Table 2##, and the standard deviation was
chosen to be 50% of its mean. Negative values were replaced with
values drawn from uniform distribution with a range between 0 and twice the mean
value. For the simulations involving individual trained neurons, the U, D, and F
parameters of these neurons were set to the values from ##TAB##1##Table 2##.</p> <p>We have carried out control experiments with current-based synapses that were not
subject to short-term plasticity (see ##SUPPL##4##Figure S5##, ##SUPPL##7##Figure S8##,
and ##SUPPL##8##Figure
S9##; successful control experiments with static current-based synapses
were also carried out for computer simulation 1, results not shown). We found
that the results of all our computer simulations also hold for static
current-based synapses.</p> <title>Model of Background Synaptic Activity</title> <p>To reproduce the background synaptic input cortical neurons receive in vivo, the
neurons in our models received an additional noise process as conductance input.
The noise process we used is a point-conductance approximation model, described
in ##REF##11744242##[26]##. According to ##REF##11744242##[26]##, this noise
process models the effect of a bombardment by a large number of synaptic inputs
in vivo, which causes membrane potential depolarization, referred to as
“high conductance” state. Furthermore, it was shown that it
captures the spectral and amplitude characteristics of the input conductances of
a detailed biophysical model of a neocortical pyramidal cell that was matched to
intracellular recordings in cat parietal cortex in vivo. The ratio of average
contributions of excitatory and inhibitory background conductances was chosen to
be 5 in accordance to experimental studies during sensory responses (see ##REF##9620800##[40]##–##REF##10816319##[42]##). In this model,
the noisy synaptic current <italic>I<sub>noise</sub></italic> in Equation 24 is
a sum of two currents:where <italic>g<sub>e</sub></italic>(<italic>t</italic>) and
<italic>g<sub>i</sub></italic>(<italic>t</italic>) are time-dependent
excitatory and inhibitory conductances. The values of the respective reversal
potentials were
<italic>E<sub>e</sub></italic> = 0 mV and
<italic>E<sub>i</sub></italic> = −75
mV. The conductances <italic>g<sub>e</sub></italic>(<italic>t</italic>) and
<italic>g<sub>i</sub></italic>(<italic>t</italic>) were modeled
according to ##REF##11744242##[26]## as a one-variable stochastic process similar
to an Ornstein-Uhlenbeck process:with mean
<italic>g<sub>e</sub></italic>
<sub>0</sub> = 0.012
<italic>µ</italic>S, noise-diffusion constant
<italic>D<sub>e</sub></italic> = 0.003
<italic>µ</italic>S and time constant
<italic>τ<sub>e</sub></italic> = 2.7
ms for the excitatory conductance, and mean
<italic>g<sub>i</sub></italic>
<sub>0</sub> = 0.057
<italic>µ</italic>S, noise-diffusion constant
<italic>D<sub>i</sub></italic> = 0.0066
<italic>µ</italic>S, and time constant
<italic>τ<sub>i</sub></italic> = 10.5
ms for the inhibitory conductance.
<italic>χ</italic>
<sub>1</sub>(<italic>t</italic>) and
<italic>χ</italic>
<sub>2</sub>(<italic>t</italic>) are Gaussian white noise
of zero mean and unit standard deviation.</p> <p>Since these processes are Gaussian stochastic processes, they can be numerically
integrated by an exact update rule:where <italic>N</italic>
<sub>1</sub>(0,1) and
<italic>N</italic>
<sub>2</sub>(0,1) are normal random numbers (zero mean,
unit standard deviation) and <italic>A<sub>e</sub></italic>,
<italic>A<sub>i</sub></italic> are amplitude coefficients given by:
</p> <title>Reward-Modulated STDP</title> <p>For the computer simulations we used the following parameters for the STDP window
function <italic>W</italic>(<italic>r</italic>):
<italic>A</italic>
<sub>+</sub> = 0.01<italic>w<sub>max</sub></italic>,
<italic>A</italic>
<sub>−</sub>/<italic>A</italic>
<sub>+</sub> = 1.05,
<italic>τ</italic>
<sub>+</sub> = <italic>τ</italic>
<sub>−</sub> = 30
ms. <italic>w<sub>max</sub></italic> denotes the hard bound of the synaptic
weight of the particular plastic synapse. Note that the parameter
<italic>A</italic>
<sub>+</sub> can be given arbitrary value in this
plasticity rule, since it can be scaled together with the reward signal, i.e.
multiplying the reward signal by some constant and dividing
<italic>A</italic>
<sub>+</sub> by the same constant results in
identical time evolution of the weight changes. We have set
<italic>A</italic>
<sub>+</sub> to be 1% of the maximum
synaptic weight.</p> <p>We used the <italic>α</italic>-function to model the eligibility trace
kernel <italic>f<sub>c</sub></italic>(<italic>t</italic>)where the time constant <italic>τ<sub>e</sub></italic>
was set to
<italic>τ<sub>e</sub></italic> = 0.4 s
in all computer simulations.</p> <p>For computer simulations 1 and 4 we performed control experiments (see ##SUPPL##2##Figure S3##,
##SUPPL##3##Figure
S4##, and ##SUPPL##6##Figure S7##) with the weight-dependent synaptic update rule proposed
in ##REF##17444756##[22]##, instead of the purely additive rule in Equation
3. We used the parameters proposed in ##REF##17444756##[22]##, i.e.
<italic>μ</italic> = 0.4,
<italic>α</italic> = 0.11,
<italic>τ</italic>
<sub>+</sub> = <italic>τ</italic>
<sub>−</sub> = 20
ms. The <italic>w</italic>
<sub>0</sub> parameter was calculated according to the
formula:  where <italic>w<sub>max</sub></italic> is the maximum synaptic
weight of the synapse.  is equal to the initial synaptic weight for the circuit
neurons, or to the mean of the distribution of the initial weights for the
trained neurons.</p> <title>Initial Weights of Trained Neurons</title> <p>The synaptic weights of excitatory synapses to the trained neurons in experiments
2–5 were initialized from a Gaussian distribution with mean
<italic>w<sub>max</sub></italic>/2. The standard deviation was set to
<italic>w<sub>max</sub></italic>/10 bounded within the range
[3<italic>w<sub>max</sub></italic>/10,7<italic>w<sub>max</sub></italic>/10].</p> <title>Software</title> <p>All computer simulations were carried out with the PCSIM software package
(<ext-link ext-link-type="uri" xlink:href="http://www.lsm.tugraz.at/pcsim">http://www.lsm.tugraz.at/pcsim</ext-link>). PCSIM is a parallel simulator
for biologically realistic neural networks with a fast c++
simulation core and a Python interface. It has been developed by Thomas
Natschläger and Dejan Pecevski. The time step of simulation was set to
0.1 ms.</p> <title>Details to Individual Computer Simulations</title> <p>For all computer simulations, both for the cortical microcircuits and readout
neurons, the same parameters values for the neuron and synapse models and the
reward-modulated STDP rule were used, as specified in the previous section
(except in computer simulation 3, where the goal was to test the theoretical
predictions for different values of the parameters). Each of the computer
simulations in this article modeled a specific task or experimental finding.
Consequently, the dependence of the reward signal on the behavior of the system
had to be modeled in a specific way for each simulation (a more detailed
discussion of the reward signal can be found in the <xref ref-type="sec" rid="s4">Discussion</xref> section). The parameters for that are given below
in separate subsections which address the individual simulations. Furthermore,
some of the remaining parameters in the experiments, i.e. the values of the
synaptic weights, the number of synapses of a neuron, number of neurons in the
circuit and the Ornstein-Uhlenbeck (OU) noise levels were chosen to achieve
different goals depending on the particular experiment. Briefly stated, these
values were tuned to achieve a certain level of firing activity in the neurons,
a suitable dynamical regime of the activity in the circuits, and a specific
ratio between amount of input the neurons receive from the input synapses and
the input generated by the noise process.</p> <p>We carried out two types of simulations: simulations of cortical microcircuits in
computer simulations 1 and 5, and training of readout neurons in computer
simulations 2, 3, 4, and 5. In the following we discuss these two types of
simulations in more detail.</p> <title>Cortical Microcircuits</title> <p>The values of the initial weights of the excitatory and inhibitory synapses for
the cortical microcircuits are given in ##TAB##2##Table 3##. All synaptic weights were bounded in
the range between 0 and twice the initial synaptic weight of the synapse.</p> <p>The cortical microcircuit was composed of 4000 neurons connected randomly with
connection probabilities described in Details to computer simulation 1. The
initial synaptic weights of the synapses and the levels of OU noise were tuned
to achieve a spontaneous firing rate of about 4.6 Hz, while maintaining an
asynchronous irregular firing activity in the circuit. 50% of all
neurons (randomly chosen, 50% excitatory and 50%
inhibitory) received downscaled OU noise (by a factor 0.2 from the model
reported in ##REF##11744242##[26]##), with the subtracted part substituted by
additional synaptic input from the circuit. The input connection probabilities
of these neurons were scaled up, so that the firing rates remain in the same
range as for the other neurons. This was done in order to observe how the
learning mechanisms work when most of the input conductance in the neuron comes
from a larger number of input synapses which are plastic, rather than from a
static noise process. The reinforced neurons were randomly chosen from this
group of neurons.</p> <p>We chose a smaller microcircuit, composed of 540 neurons, for the computer
simulation 5 in order to be able to perform a large number of training trials.
The synaptic weights in this smaller circuit were chosen (see ##TAB##2##Table 3##) to achieve an
appropriate level of firing activity in the circuit that is modulated by the
external input. The circuit neurons had injected an Ornstein-Uhlenbeck (OU)
noise multiplied by 0.4 in order to emulate the background synaptic activity in
neocortical neurons in vivo, and test the learning in a more biologically
realistic settings. This produced significant trial-to-trial variability in the
circuit response (see ##FIG##9##Figure
10D##). A lower value of the noise level could also be used without
affecting the learning, whereas increasing the amount of injected noise would
slowly deteriorate the information that the circuit activity maintains about the
injected inputs, resulting in a decline of the learning performance.</p> <title>Readout Neurons</title> <p>The maximum values of the synaptic weights of readout neurons for computer
simulations 2, 4, and 5, together with the number of synapses of the neurons,
are given in ##TAB##3##Table 4##.</p> <p>The neuron in computer simulation 2 had 100 synapses. We chose 200 synapses for
the neuron in computer simulation 4, in order to improve the learning
performance. Such improvement of the learning performance for larger numbers of
synapses is in accordance with our theoretical analysis (see Equation 17), since
for learning the classification of temporal patterns the temporal variation of
the voltage of the postsynaptic membrane turns out to be of critical importance
(see the discussion after Equation 17).
This temporal variation depends less on the shape of a single EPSP and more on
the temporal pattern of presynaptic firing when the number of synapses is
increased. In computer simulation 5 the readout neuron received inputs from all
432 excitatory neurons in the circuit. The synaptic weights were chosen in
accordance with the number of synapses in order to achieve a firing rate
suitable for the particular task, and to balance the synaptic input and the
noise injections in the neurons.</p> <p>For the pattern discrimination task (computer simulation 4) and the speech
recognition task (computer simulation 5), the amount of noise had to be chosen
to be high enough to achieve sufficient variation of the membrane potential from
trial to trial near the firing threshold, and low enough so that it would not
dominate the fluctuations of the membrane potential. In the experiment where the
exact spike times were rewarded (computer simulation 2), the noise had a
different role. As described in the <xref ref-type="sec" rid="s2">Results</xref>
section, there the noise effectively controls the amount of depression. If the
noise (and therefore the depression) is too weak,
<italic>w</italic>* = 0 synapses do not
converge to 0. If the noise is too strong,
<italic>w</italic>* = <italic>w<sub>max</sub></italic>
synapses do not converge to <italic>w<sub>max</sub></italic>. To achieve the
desired learning result, the noise level should be in a range where it reduces
the correlations of the synapses with
<italic>w</italic>* = 0 so that the
depression of STDP will prevail, but at the same time is not strong enough to do
the same for the other group of synapses with
<italic>w</italic>* = <italic>w<sub>max</sub></italic>,
since they have stronger pre-before-post correlations. For our simulations, we
have set the noise level to the full amount of OU noise.</p> <title>Details to Computer Simulation 1: Model for Biofeedback Experiment</title> <p>The cortical microcircuit model consisted of 4000 neurons with twenty percent of
the neurons randomly chosen to be inhibitory, and the others excitatory. The
connections between the neurons were created randomly, with different
connectivity probabilities depending on whether the postsynaptic neuron received
the full amount of OU noise, or downscaled OU noise with an additional
compensatory synaptic input from the circuit. For neurons in the latter
sub-population, the connection probabilities were
<italic>p<sub>ee</sub></italic> = 0.02,
<italic>p<sub>ei</sub></italic> = 0.02,
<italic>p<sub>ie</sub></italic> = 0.024
and <italic>p<sub>ii</sub></italic> = 0.016
where the ee, ei, ie, ii indices designate the type of the presynaptic and
postsynaptic neurons (e = excitatory or
i = inhibitory). For the other neurons the
corresponding connection probabilities were downscaled by 0.4. The resulting
firing rates and correlations for both types of excitatory neurons are plotted
in ##SUPPL##0##Figure
S1## and ##SUPPL##1##Figure S2##.</p> <p>The shape of the reward kernel
<italic>ε<sub>r</sub></italic>(<italic>t</italic>) was chosen as a
difference of two <italic>α</italic>-functionsone positive <italic>α</italic>-pulse with a peak at 0.4
sec after the corresponding spike, and one long-tailed negative
<italic>α</italic>-pulse which makes sure that the integral over the
reward kernel is zero. The parameters for the reward kernel were , , , , and
<italic>d<sub>r</sub></italic> = 0.2 s, which
produced a peak value of the reward pulse 0.4 s after the spike that caused
it.</p> <title>Details to Computer Simulation 2: Learning Spike Times</title> <p>We used the following function for the reward kernel
<italic>κ</italic>(<italic>r</italic>)where  and  are positive scaling constants,  and  define the shape of the two double-exponential functions the
kernel is composed of, and <italic>t<sub>κ</sub></italic> defines the
offset of the zero-crossing from the origin. The parameter values used in our
simulations were , , ,  and
<italic>t<sub>κ</sub></italic> = −1
ms. The reward delay was equal to
<italic>d<sub>r</sub></italic> = 0.4 s.</p> <title>Details to Computer Simulation 3: Testing the Analytically Derived Conditions</title> <p>We used a linear Poisson neuron model as in the theoretical analysis with static
synapses and exponentially decaying postsynaptic responses . The neuron had 100 excitatory synapses, except in experiment
#6, where we used 200 synapses. In all experiments the target neuron received
additional 10 excitatory synapses with weights set to
<italic>w<sub>max</sub></italic>. The input spike trains were Poisson processes
with a constant rate of
<italic>r<sub>pre</sub></italic> = 6 Hz, except
in experiment # 6 where the rate was
<italic>r<sub>pre</sub></italic> = 3 Hz. The
weights of the target neuron were set to  for 0≤<italic>i</italic>&lt;50 and  for 50≤<italic>i</italic>&lt;100.</p> <p>The time constants of the reward kernel were , whereas  had different values in different experiments (reported in
##TAB##0##table 1##). The value of
<italic>t<sub>κ</sub></italic> was always set to an optimal
value such that the . The time constant
<italic>τ</italic>
<sub>−</sub> of the negative part of the
STDP window function <italic>W</italic>(<italic>r</italic>) was set to
<italic>τ</italic>
<sub>+</sub>. The reward signal was
delayed by
<italic>τ<sub>d</sub></italic> = 0.4 s.
The simulations were performed for varying durations of simulated biological
time (see the <italic>t<sub>sim</sub></italic>-column in ##TAB##0##Table 1##).</p> <title>Details to Computer Simulation 4: Learning Pattern Classification</title> <p>We used the reward signal from Equation 16, with an
<italic>α</italic>-function for the reward kernel , and the reward delay <italic>d<sub>r</sub></italic> set to
300 ms. The amplitudes of the positive and negative pulses were
<italic>α<sub>P</sub></italic> = −<italic>α<sub>N</sub></italic> = 1.435
and the time constant of the reward kernel was
<italic>τ</italic> = 100 ms.</p> <title>Details to Computer Simulation 5: Training a Readout Neuron with
Reward-Modulated STDP To Recognize Isolated Spoken Digits</title> <title>Spike representations of speech utterances</title> <p>The speech utterances were preprocessed by the cochlea model described in
##UREF##6##[43]##, which captures the filtering properties of
the cochlea and hair cells in the human inner ear. The resulting analog
signals were encoded by spikes with the BSA spike encoding algorithm
described in ##UREF##7##[44]##. We used the same preprocessing to
generate the spikes as in ##UREF##8##[45]##. The spike
representations had a duration of about 400 ms and 20 input channels. The
input channels were connected topographically to the cortical microcircuit
model. The neurons in the circuit were split into 20 disjunct subsets of 27
neurons, and each input channel was connected to the 27 neurons in its
corresponding subsets. The readout neuron was trained with 20 different
spike inputs to the circuit, where 10 of them resulted from utterances of
digit “one”, and the other 10 resulted from utterances
of digit “two” by the same speaker.</p> <title>Training procedure</title> <p>We performed 2000 training trials, where for each trial a spike
representation of a randomly chosen utterance out of 10 utterances for one
digit was injected into the circuit. The digit changed from trial to trial.
Whenever the readout neuron spiked during the presentation of an utterance
of digit “two”, a positive pulse was generated in the
reward signal, and accordingly, for utterances of digit
“one”, a negative pulse in the reward was generated. We
used the reward signal from Equation 16. The amplitudes of the positive and
negative pulses were
<italic>α<sub>P</sub></italic> = −<italic>α<sub>N</sub></italic> = 0.883.
The time constant of the reward kernel
<italic>ε<sub>r</sub></italic>(<italic>r</italic>) was
<italic>τ</italic> = 100 ms.
The pulses in the reward were delayed
<italic>d<sub>r</sub></italic> = 300 ms
from the spikes that caused them.</p> <title>Cortical microcircuit details</title> <p>The cortical microcircuit model consisted of 540 neurons with twenty percent
of the neurons randomly chosen to be inhibitory, and the others excitatory.
The recurrent connections in the circuit were created randomly with a
connection probability of 0.1. Long-term plasticity was not modeled in the
circuit synapses.</p> <p>The synapses for the connections from the input neurons to the circuit
neurons were static, current based with axon conduction delay of 1 ms, and
exponentially decaying PSR with time constant
<italic>τ<sub>e</sub></italic> = 3
ms and amplitude
<italic>w<sub>input</sub></italic> = 0.715
nA.</p>
<title>Results</title> <p>We first give a precise definition of the learning rule in Equation 1 for
reward-modulated STDP. The standard rule for STDP, which specifies the change
<italic>W</italic>(Δ<italic>t</italic>) of the synaptic weight of an
excitatory synapse in dependence on the time difference
Δ<italic>t</italic> = <italic>t<sub>post</sub></italic>−<italic>t<sub>pre</sub></italic>
between the firing times <italic>t<sub>pre</sub></italic> and
<italic>t<sub>post</sub></italic> of the pre- and postsynaptic neuron, is based
on numerous experimental data (see ##REF##11127835##[1]##). It is commonly modeled by a so-called learning
curve of the formwhere the positive constants <italic>A</italic>
<sub>+</sub>
and <italic>A</italic>
<sub>−</sub> scale the strength of potentiation and
depression respectively, and <italic>τ</italic>
<sub>+</sub> and
<italic>τ</italic>
<sub>−</sub> are positive time constants
defining the width of the positive and negative learning window. The resulting
weight change at time <italic>t</italic> of synapse <italic>ji</italic> for a
presynaptic spike train  and a postsynaptic spike train  is usually modeled ##UREF##2##[21]## by the instantaneous
application of this learning rule to all spike pairings with the second spike at
time <italic>t</italic>
The spike train of a neuron <italic>i</italic> which fires action
potentials at times , , ,… is formalized here by a sum of Dirac delta functions .</p> <p>The model analyzed in this article is based on the assumption that positive and
negative weight changes suggested by STDP for all pairs of pre- and postsynaptic
spikes at synapse <italic>ji</italic> (according to the two integrals in Equation 3)
are collected in an eligibility trace
<italic>c<sub>ji</sub></italic>(<italic>t</italic>) at the site of the synapse. The
contribution to <italic>c<sub>ij</sub></italic>(<italic>t</italic>) of all spike
pairings with the second spike at time
<italic>t</italic>−<italic>s</italic> is modeled for
<italic>s</italic>&gt;0 by a function
<italic>f<sub>c</sub></italic>(<italic>s</italic>) (see ##FIG##0##Figure 1A##); the time scale of the eligibility
trace is assumed in this article to be on the order of seconds. Hence the value of
the eligibility trace of synapse <italic>ji</italic> at time <italic>t</italic> is
given bysee ##FIG##0##Figure 1B##.
The actual weight change  at time <italic>t</italic> for reward-modulated STDP is the
product
<italic>c<sub>ij</sub></italic>(<italic>t</italic>)·<italic>d</italic>(<italic>t</italic>)
of the eligibility trace with the reward signal
<italic>d</italic>(<italic>t</italic>) as defined by Equation 1. Since this simple
model can in principle lead to unbounded growth of weights, we assume that weights
are clipped at the lower boundary value 0 and an upper boundary
<italic>w<sub>max</sub></italic>.</p> <p>The network dynamics of a simulated recurrent network of spiking neurons where all
connections between excitatory neurons are subject to STDP is quite sensitive to the
particular STDP-rule that is used. Therefore we have carried out our network
simulations not only with the additive STDP-rule in Equation 3, whose effect can be
analyzed theoretically, but also with the more complex rule proposed in ##REF##17444756##[22]##
(which was fitted to experimental data from hippocampal neurons in culture ##REF##9852584##[23]##), where the
magnitude of the weight change depends on the current value of the weight. An
implementation of this STDP-rule (with the parameters proposed in ##REF##17444756##[22]##)
produced in our network simulations of the biofeedback experiment (computer
simulation 1) as well as for learning pattern classification (computer simulation 4)
qualitatively the same result as the rule in Equation 3.</p> <title>Theoretical Analysis of the Resulting Weight Changes</title> <p>In this section, we derive a learning equation for reward-modulated STDP. This
learning equation relates the change of a synaptic weight
<italic>w<sub>ji</sub></italic> over some sufficiently long time interval
<italic>T</italic> to statistical properties of the joint distribution of
the reward signal <italic>d</italic>(<italic>t</italic>) and pre- and
postsynaptic firing times, under the assumption that the weight and correlations
between pre- and postsynaptic spike times are slowly varying in time. We treat
spike times as well as the reward signal <italic>d</italic>(<italic>t</italic>)
as stochastic variables. This mathematical framework allows us to derive the
expected weight change over some time interval <italic>T</italic> (see ##UREF##2##[21]##),
with the expectation taken over realizations of the stochastic input- and output
spike trains as well as stochastic realizations of the reward signal, denoted by
the ensemble average 〈·〉<italic><sub>E</sub></italic>
where we used the abbreviation . If synaptic plasticity is sufficiently slow, synaptic weights
integrate a large number of small changes. In this case, the weight
<italic>w<sub>ji</sub></italic> can be approximated by its average
〈<italic>w<sub>ji</sub></italic>〉<italic><sub>E</sub></italic> (it is “self-averaging”, see ##UREF##2##[21]##). We can thus
drop the expectation on the left hand side of Equation 5 and write it as . Using Equation 1, this yields (see <xref ref-type="sec" rid="s3">Methods</xref>)This formula contains the <italic>reward correlation</italic> for
synapse <italic>ji</italic>
which is the average reward at time <italic>t</italic> given a
presynaptic spike at time
<italic>t</italic>−<italic>s</italic>−<italic>r</italic>
and a postsynaptic spike at time
<italic>t</italic>−<italic>s</italic>. The joint firing rate
<italic>ν<sub>ji</sub></italic>(<italic>t</italic>,<italic>r</italic>) = 〈<italic>S<sub>j</sub></italic>(<italic>t</italic>)<italic>S<sub>i</sub></italic>(<italic>t</italic>−<italic>r</italic>)〉<italic><sub>E</sub></italic> describes correlations between spike timings of neurons
<italic>j</italic> and <italic>i</italic>, i.e., it is the probability density
for the event that neuron <italic>i</italic> fires an action potential at time
<italic>t</italic>−<italic>r</italic> and neuron
<italic>j</italic> fires an action potential at time <italic>t</italic>. For
synapses subject to reward-modulated STDP, changes in efficacy are obviously
driven by co-occurrences of spike pairings and rewards within the time scale of
the eligibility trace. Equation 6 clarifies how the expected weight change
depends on how the correlations between the pre- and postsynaptic neurons
correlate with the reward signal.</p> <p>If one assumes for simplicity that the impact of a spike pair on the eligibility
trace is always triggered by the postsynaptic spike, one gets a simpler equation
(see <xref ref-type="sec" rid="s3">Methods</xref>)The assumption introduces a small error for post-before-pre spike
pairs, because for a reward signal that arrives at some time
<italic>d<sub>r</sub></italic> after the pairing, the weight update will be
proportional to <italic>f<sub>c</sub></italic>(<italic>d<sub>r</sub></italic>)
instead of
<italic>f<sub>c</sub></italic>(<italic>d<sub>r</sub></italic>+<italic>r</italic>).
The approximation is justified if the temporal average is performed on a much
longer time scale than the time scale of the learning window, the effect of each
pre-post spike pair on the reward signal is delayed by an amount greater than
the time scale of the learning window, and <italic>f<sub>c</sub></italic>
changes slowly compared to the time scale of the learning window (see <xref ref-type="sec" rid="s3">Methods</xref> for details). For the analyzes
presented in this article, the simplified Equation 8 is a good approximation for
the learning dynamics. Equation 8 is a generalized version of the STDP learning
equation  in ##UREF##2##[21]## that includes the impact of the reward
correlation weighted by the eligibility function. To see the relation between
standard STDP and reward-modulated STDP, consider a constant reward signal
<italic>d</italic>(<italic>t</italic>) = <italic>d</italic>
<sub>0</sub>.
Then also the reward correlation is constant and given by
<italic>D</italic>(<italic>t</italic>,<italic>s</italic>,<italic>r</italic>) = <italic>d</italic>
<sub>0</sub>.
We recover the standard STDP learning equation scaled by
<italic>d</italic>
<sub>0</sub> if the eligibility function is an
instantaneous delta-pulse
<italic>f<sub>c</sub></italic>(<italic>s</italic>) = <italic>δ</italic>(<italic>s</italic>).
Furthermore, if the statistics of the reward signal
<italic>d</italic>(<italic>t</italic>) is time-independent and independent from
the pre- and postsynaptic spike statistics of some synapse <italic>ji</italic>,
then the reward correlation is given by
<italic>D<sub>ji</sub></italic>(<italic>t</italic>,<italic>s</italic>,<italic>r</italic>) = 〈<italic>d</italic>(<italic>t</italic>)〉<italic><sub>E</sub></italic> = <italic>d</italic>
<sub>0</sub> for
some constant <italic>d</italic>
<sub>0</sub>. Then, the weight change for
synapse <italic>ji</italic> is . The temporal average of the joint firing rate
〈<italic>ν<sub>ji</sub></italic>(<italic>t</italic>−<italic>s</italic>,<italic>r</italic>〉<italic><sub>T</sub></italic> is thus filtered by the eligibility trace. We assumed in the preceding
analysis that the temporal average is taken over some long time interval
<italic>T</italic>. If the time scale of the eligibility trace is much
smaller than this time interval <italic>T</italic>, then the weight change is
approximately , and the weight <italic>w<sub>ji</sub></italic> will change
according to standard STDP scaled by a constant proportional to the mean reward
and the integral over the eligibility function. In the remainder of this
article, we will always use the smooth time-averaged weight change , but for brevity, we will drop the angular brackets and simply
write .</p> <p>The learning Equation 8 provides the mathematical basis for our following
analyses. It allows us to determine synaptic weight changes if we can describe a
learning situation in terms of reward correlations and correlations between pre-
and postsynaptic spikes.</p> <title>Application to Models for Biofeedback Experiments</title> <p>We now apply the preceding analysis to the biofeedback experiment of ##REF##4196269##[17]## that
were described in the introduction. These experiments pose the challenge to
explain how learning mechanisms in the brain can detect and exploit correlations
between rewards and the firing activity of one or a few neurons within a large
recurrent network of neurons (the credit assignment problem), without changing
the overall function or dynamics of the circuit.</p> <p>We show that this phenomenon can in principle be explained by reward-modulated
STDP. In order to do that, we define a model for the experiment which allows us
to formulate an equation for the reward signal
<italic>d</italic>(<italic>t</italic>). This enables us to calculate synaptic
weight changes for this particular scenario. We consider as model a recurrent
neural circuit where the spiking activity of one neuron <italic>k</italic> is
recorded by the experimenter (Experiments where two neurons are recorded and
reinforced were also reported in ##REF##4196269##[17]##. We tested this case in computer simulations
(see ##FIG##1##Figure 2##) but did not
treat it explicitly in our theoretical analysis). We assume that in the monkey
brain a reward signal <italic>d</italic>(<italic>t</italic>) is produced which
depends on the visual feedback (through an illuminated meter, whose pointer
deflection was dependent on the current firing rate of the randomly selected
neuron <italic>k</italic>) as well as previously received liquid rewards, and
that this signal <italic>d</italic>(<italic>t</italic>) is delivered to
<italic>all</italic> synapses in large areas of the brain. We can formalize
this scenario by defining a reward signal which depends on the spike rate of the
arbitrarily selected neuron <italic>k</italic> (see ##FIG##2##Figure 3A and 3B##). More precisely, a reward
pulse of shape <italic>ε<sub>r</sub></italic>(<italic>r</italic>) (the
reward kernel) is produced with some delay <italic>d<sub>r</sub></italic> every
time the neuron <italic>k</italic> produces an action potentialNote that
<italic>d</italic>(<italic>t</italic>) = <italic>h</italic>(<italic>t</italic>)−<italic>h̅</italic>
is defined in Equation 1 as a signal with zero mean. In order to satisfy this
constraint, we assume that the reward kernel
<italic>ε<sub>r</sub></italic> has zero mass, i.e., . For the analysis, we use the linear Poisson neuron model
described in <xref ref-type="sec" rid="s3">Methods</xref>. The mean weight
change for synapses to the reinforced neuron <italic>k</italic> is then
approximately (see <xref ref-type="sec" rid="s3">Methods</xref>)This equation describes STDP with a learning rate proportional to . The outcome of the learning session will strongly depend on
this integral and thus on the form of the reward kernel
<italic>ε<sub>r</sub></italic>. In order to reinforce high firing
rates of the reinforced neuron we have chosen a reward kernel with a positive
bump in the first few hundred milliseconds, and a long negative tail afterwards.
##FIG##2##Figure 3C## shows the
functions <italic>f<sub>c</sub></italic> and
<italic>ε<sub>r</sub></italic> that were used in our computer model, as
well as the product of these two functions. One sees that the integral over the
product is positive and according to Equation 10 the synapses to the reinforced
neuron are subject to STDP. This does not guarantee an increase of the firing
rate of the reinforced neuron. Instead, the changes of neuronal firing will
depend on the statistics of the inputs. In particular, the weights of synapses
to neuron <italic>k</italic> will not increase if that neuron does not fire
spontaneously. For uncorrelated Poisson input spike trains of equal rate, the
firing rate of a neuron trained by STDP stabilizes at some value which depends
on the input rate (see ##REF##10966623##[24]##,##REF##11705408##[25]##). However, in
comparison to the low spontaneous firing rates observed in the biofeedback
experiment ##REF##4196269##[17]##, the stable firing rate under STDP can be much
higher, allowing for a significant rate increase. It was shown in ##REF##4196269##[17]## that
also low firing rates of a single neuron can be reinforced. In order to model
this, we have chosen a reward kernel with a negative bump in the first few
hundred milliseconds, and a long positive tail afterwards, i.e. we inverted the
kernel used above to obtain a negative integral . According to Equation 10 this leads to anti-STDP where not
only inputs to the reinforced neuron which have low correlations with the output
are depressed (because of the negative integral of the learning window), but
also those which are causally correlated with the output. This leads to a quick
firing rate decrease at the reinforced neuron.</p> <p>The mean weight change of synapses to non-reinforced neurons
<italic>j</italic>≠<italic>k</italic> is given bywhere
<italic>ν<sub>j</sub></italic>(<italic>t</italic>) = 〈<italic>S<sub>j</sub></italic>(<italic>t</italic>)〉<italic><sub>E</sub></italic> is the instantaneous firing rate of neuron <italic>j</italic> at time
<italic>t</italic>. This equation indicates that a non-reinforced neuron is
trained by STDP with a learning rate proportional to its correlation with the
reinforced neuron given by
<italic>ν<sub>kj</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′,<italic>s</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′)/<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>).
In fact, it was noted in ##REF##4196269##[17]## that neurons nearby the reinforced neuron
tended to change their firing rate in the same direction. This observation might
be explained by putative correlations of the recorded neuron with nearby
neurons. On the other hand, if a neuron <italic>j</italic> is uncorrelated with
the reinforced neuron <italic>k</italic>, we can decompose the joint firing rate
into
<italic>ν<sub>kj</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′,<italic>s</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′) = <italic>ν<sub>k</sub></italic>(<italic>t</italic>−<italic>d<sub>r</sub></italic>−<italic>r</italic>′)<italic>ν<sub>j</sub></italic>(<italic>t</italic>−<italic>s</italic>).
In this case, the learning rate for synapse <italic>ji</italic> is approximately
zero (see <xref ref-type="sec" rid="s3">Methods</xref>). This ensures that most
neurons in the circuit keep a constant firing rate, in spite of continuous
weight changes according to reward-modulated STDP.</p> <p>Altogether we see that the weights of synapses to the reinforced neuron
<italic>k</italic> can only change if there is spontaneous activity in the
network, so that in particular also this neuron <italic>k</italic> fires
spontaneously. On the other hand the spontaneous network activity should not
consist of repeating large-scale spatio-temporal firing patterns, since that
would entail correlations between the firing of neuron <italic>k</italic> and
other neurons <italic>j</italic>, and would lead to similar changes of synapses
to these other neurons <italic>j</italic>. Apart from these requirements on the
spontaneous network activity, the preceding theoretical results predict that
stability of the circuit is preserved, while the neuron which is causally
related to the reward signal is trained by STDP, if  is positive.</p> <title>Computer Simulation 1: Model for Biofeedback Experiment</title> <p>We tested these theoretical predictions through computer simulations of a generic
cortical microcircuit receiving a reward signal which depends on the firing of
one arbitrarily chosen neuron <italic>k</italic> from the circuit (reinforced
neuron). The circuit was composed of 4000 LIF neurons, with 3200 being
excitatory and 800 inhibitory, interconnected randomly by 228954 conductance
based synapses with short term dynamics (All computer simulations were also
carried out as a control with static current based synapses, see <xref ref-type="sec" rid="s3">Methods</xref> and Suppl.). In addition to the
explicitly modeled synaptic connections, conductance noise (generated by an
Ornstein-Uhlenbeck process) was injected into each neuron according to data from
##REF##11744242##[26]##, in order to model synaptic background activity
of neocortical neurons in-vivo (More precisely, for 50% of the
excitatory neurons the amplitude of the noise injection was reduced to
20%, and instead their connection probabilities from other excitatory
neurons were chosen to be larger, see <xref ref-type="sec" rid="s3">Methods</xref> and ##SUPPL##0##Figure S1## and ##SUPPL##1##Figure S2##
for details. The reinforced neuron had to be chosen from the latter population,
since reward-modulated STDP does not work properly if the postsynaptic neuron
fires too often because of directly injected noise). This background noise
elicited spontaneous firing in the circuit at about 4.6 Hz. Reward-modulated
STDP was applied continuously to all synapses which had excitatory presynaptic
and postsynaptic neurons, and all these synapses received the same reward
signal. The reward signal was modeled according to Equation 9. ##FIG##2##Figure 3C## shows one reward
pulse caused by a single postsynaptic spike at time
<italic>t</italic> = 0 with the parameters used
in the experiment. For several postsynaptic spikes, the amplitude of the reward
signal follows the firing rate of the reinforced neuron, see ##FIG##2##Figure 3B##.</p> <p>This model was simulated for 20 minutes of biological time. ##FIG##3##Figure 4A, 4B, and 4D## show that the firing
rate of the reinforced neuron increases within a few minutes (like in the
experiment of ##REF##4196269##[17]##), while the firing rates of the other neurons
remain largely unchanged. The increase of weights to the reinforced neuron shown
in ##FIG##3##Figure 4C## can be
explained by the correlations between its presynaptic and postsynaptic spikes
shown in panel E. This panel shows that pre-before-post spike pairings (black
curve) are in general more frequent than post-before-pre spike pairings. The
reinforced neuron increases its rate from around 4 Hz to 12 Hz, which is
comparable to the measured firing rates in ##REF##17571943##[15]## before and after
learning.</p> <p>In Figure 9 of ##REF##4196269##[17]## and
Figure 1 of ##REF##17234689##[19]## the
results of another experiment were reported where the activity of two adjacent
neurons was recorded, and high firing rates of the first neuron and low firing
rates of the second neuron were reinforced simultaneously. This kind of
differential reinforcement resulted in an increase and decrease of the firing
rates of the two neurons correspondingly. We implemented this type of
reinforcement by letting the reward signal in our model depend on the spikes of
the two randomly chosen neurons (we refer to these neurons as neuron A and
neuron B), i.e. , where  is the component that positively rewards spikes of neuron A,
and  negatively rewards spikes of neuron B. Both parts of the
reward signal,  and , were defined as in Equation 9 for the corresponding neuron.
For  we used the reward kernel
<italic>ε<sub>r</sub></italic> as defined in Equation 29, whereas for  we used
<italic>ε<sub>r</sub></italic>
<sub>−</sub> = −<italic>ε<sub>r</sub></italic>
(note that the integral over
<italic>ε<sub>r</sub></italic>
<sub>−</sub> is still zero).
At the middle of the simulation (simulation time
<italic>t</italic> = 10 min), we changed the
direction of the reinforcements by negatively rewarding the firing of neuron A
and positively rewarding the firing of neuron B (i.e., ). The results are summarized in ##FIG##1##Figure 2##. With a reward signal modeled in
this way, we were able to independently increase and decrease the firing rates
of the two neurons according to the reinforcements, while the firing rates of
the other neurons remained unchanged. Changing the type of reinforcement during
the simulation from positive to negative for neuron A and from negative to
positive for neuron B resulted in a corresponding shift in their firing rate
change in the direction of the reinforcement.</p> <p>The dynamics of a network where STDP is applied to all synapses between
excitatory neurons is quite sensitive to the specific choice of the STDP-rule.
The preceding theoretical analysis (see Equations 10 and 11) predicts that
reward-modulated STDP affects in the long run only those excitatory synapses
where the firing of the postsynaptic neuron is correlated with the reward
signal. In other words: the reward signal gates the effect of STDP in a
recurrent network, and thereby can keep the network within a given dynamic
regime. This prediction is confirmed qualitatively by the two panels of ##FIG##3##Figure 4A##, which show that
even after all excitatory synapses in the recurrent network have been subject to
20 minutes (in simulated biological time) of reward-modulated STDP, the network
stays within the asynchronous irregular firing regime. It is also confirmed
quantitatively through ##FIG##4##Figure
5##. These figures show results for the simple additive version of STDP
(according to Equation 3). Very similar results (see ##SUPPL##2##Figure S3##
and ##SUPPL##3##Figure
S4##) arise from an application of the more complex STDP-rule proposed in
##REF##17444756##[22]## where the weight-change depends on the current
weight value.</p> <title>Rewarding Spike-Times</title> <p>The preceding model for the biofeedback experiment of Fetz and Baker focused on
learning of firing rates. In order to explore the capabilities and limitations
of reward-modulated STDP in contexts where the temporal structure of spike
trains matters, we investigated another reinforcement learning scenario where a
neuron should learn to respond with particular temporal spike patterns. We first
apply analytical methods to derive conditions under which a neuron subject to
reward-modulated STDP can achieve this.</p> <p>In this model, the reward signal <italic>d</italic>(<italic>t</italic>) is given
in dependence on how well the output spike train  of a neuron <italic>j</italic> matches some rather arbitrary
spike train <italic>S</italic>* (which might for example represent spike
output from some other brain structure during a developmental phase).
<italic>S</italic>* is produced by a neuron
<italic>μ</italic>* that receives the same <italic>n</italic>
input spike trains
<italic>S</italic>
<sub>1</sub>,…,<italic>S<sub>n</sub></italic> as
the trained neuron <italic>j</italic>, with some arbitrarily chosen weights , . But in addition the neuron
<italic>μ</italic>* receives
<italic>n</italic>′−<italic>n</italic> further spike
trains
<italic>S<sub>n</sub></italic>
<sub>+1</sub>,…,<italic>S<sub>n</sub></italic>
<sub>′</sub>
with weights . The setup is illustrated in ##FIG##5##Figure 6A##. It provides a generic
reinforcement learning scenario, when a quite arbitrary (and not perfectly
realizable) spike output is reinforced, but simultaneously the performance of
the learner can be evaluated clearly according to how well its weights
<italic>w<sub>j</sub></italic>
<sub>1</sub>,…,<italic>w<sub>jn</sub></italic>
match those of the neuron <italic>μ</italic>* for those
<italic>n</italic> input spike trains which both of them have in common. The
reward <italic>d</italic>(<italic>t</italic>) at time <italic>t</italic> depends
in this task on both the timing of action potentials of the trained neuron and
spike times in the target spike train <italic>S</italic>*where the function <italic>κ</italic>(<italic>r</italic>)
with  describes how the reward signal depends on the time difference
<italic>r</italic> between a postsynaptic spike and a target spike, and
<italic>d<sub>r</sub></italic>&gt;0 is the delay of the reward.</p> <p>Our theoretical analysis (see <xref ref-type="sec" rid="s3">Methods</xref>)
predicts that under the assumption of constant-rate uncorrelated Poisson input
statistics this reinforcement learning task can be solved by reward-modulated
STDP for arbitrary initial weights if three constraints are fulfilled:

The following parameters occur in these equations:
<italic>ν</italic>* is the output rate of neuron
<italic>μ</italic>*,  is the minimal output rate,  is the maximal output rate of the trained neuron,  is the integral over the eligibility trace,  is the integral over the STDP learning curve (see Equation 2),  is the convolution of the reward kernel with the shape of the
postsynaptic potential (PSP) <italic>ε</italic>(<italic>s</italic>), and  is the integral over the PSP weighted by the learning window.</p> <p>If these inequalities are fulfilled and input rates are larger than zero, then
the weight vector of the trained neuron converges on average from any initial
weight vector to <bold>w</bold>* (i.e., it mimics the weight
distribution of neuron <italic>μ</italic>* for those
<italic>n</italic> inputs which both have in common). To get an intuitive
understanding of these inequalities, we first examine the idea behind Constraint
13. This constraint assures that weights of synapses <italic>i</italic> with  decay to zero in expectation. First note that input spikes
from a spike train <italic>S<sub>i</sub></italic> with  have no influence on the target spike train
<italic>S</italic>*. In the linear Poisson neuron model, this leads to
weight changes similar to STDP which can be described by two terms. First, all
synapses are subject to depression stemming from the negative part of the
learning curve <italic>W</italic> and random pre-post spike pairs. This weight
change is bounded from below by  for some positive constant <italic>α</italic>. On the
other hand, the positive influence of input spikes on postsynaptic firing leads
to potentiation of the synapse bounded from above by . Hence the weight decays to zero if , leading to Inequality 13. For synapses <italic>i</italic>
with , there is an additional drive, since each presynaptic spike
increases the probability of a closely following spike in the target spike train
<italic>S</italic>*. Therefore, the probability of a delayed reward
signal after a presynaptic spike is larger. This additional drive leads to
positive weight changes if Inequalities 14 and 15 are fulfilled (see <xref ref-type="sec" rid="s3">Methods</xref>).</p> <p>Note that also for the learning of spike times spontaneous spikes (which might be
regarded as “noise”) are important, since they may lead to
reward signals that can be exploited by the learning rule. It is obvious that in
reward-modulated STDP, a silent neuron cannot recover from its silent state,
since there will be no spikes which can drive STDP. But in addition, Condition
13 shows that in this learning scenario, the minimal output rate —which increases with increasing noise—has
to be larger than some positive constant, such that depression is strong enough
to weaken synapses if needed. On the other hand, if the noise is too strong also
synapses <italic>i</italic> with
<italic>w<sub>i</sub></italic> = <italic>w<sub>max</sub></italic>
will be depressed and may not converge correctly. This can happen when the
increased noise leads to a maximal postsynaptic rate  such that Constraints 14 and 15 are not satisfied anymore.</p> <p>Conditions 13–15 also reveal how parameters of the model influence the
applicability of this setup. For example, the eligibility trace enters the
equations only in the form of its integral and its value at the reward delay in
Equation 15. In fact, the exact shape of the eligibility trace is not important.
The important property of an ideal eligibility trace is that it is high at the
reward delay and low at other times as expressed by the fraction in Condition
15. Interestingly, the formulas also show that one has quite some freedom in
choosing the form of the STDP window, as long as the reward kernel
<italic>ε<sub>κ</sub></italic> is adjusted accordingly.
For example, instead of a standard STDP learning window <italic>W</italic> with
<italic>W</italic>(<italic>r</italic>)≥0 for
<italic>r</italic>&gt;0 and <italic>W</italic>(<italic>r</italic>)≤0
for <italic>r</italic>&lt;0 and a corresponding reward kernel
<italic>κ</italic>, one can use a reversed learning window
<italic>W</italic>′ defined by
<italic>W</italic>′(<italic>r</italic>)≡<italic>W</italic>(−<italic>r</italic>)
and a reward kernel <italic>κ</italic>′ such that
<italic>ε<sub>κ</sub></italic>
<sub>′</sub>(<italic>r</italic>) = <italic>ε<sub>κ</sub></italic>(−<italic>r</italic>).
If Condition 15 is satisfied for <italic>W</italic> and
<italic>κ</italic>, then it is also satisfied for
<italic>W</italic>′ and <italic>κ′</italic> (and in
most cases also Condition 14 will be satisfied). This reflects the fact that in
reward modulated STDP the learning window defines the weight changes in
combination with the reward signal.</p> <p>For a given STDP learning window, the analysis reveals what reward kernels
<italic>κ</italic> are suitable for this learning setup. From
Condition 15, we can deduce that the integral over <italic>κ</italic>
should be small (but positive), whereas the integral  should be large. Hence, for a standard STDP learning window
<italic>W</italic> with <italic>W</italic>(<italic>r</italic>)≥0 for
<italic>r</italic>&gt;0 and
<italic>W</italic>(<italic>r</italic>)≤0 for <italic>r</italic>&lt;0,
the convolution
<italic>ε<sub>κ</sub></italic>(<italic>r</italic>) of the reward
kernel with the PSP should be positive for <italic>r</italic>&gt;0 and
negative for <italic>r</italic>&lt;0. In the computer simulation we used a
simple kernel depicted in ##FIG##5##Figure
6B##, which satisfies the aforementioned constraints. It consists of two
double-exponential functions, one positive and one negative, with a zero
crossing at some offset <italic>t<sub>κ</sub></italic> from the origin.
The optimal offset <italic>t<sub>κ</sub></italic> is always negative and
in the order of several milliseconds for usual PSP-shapes
<italic>ε</italic>. We conclude that for successful learning in this
scenario, a positive reward should be produced if the neuron spikes around the
target spike or somewhat later, and a negative reward should be produced if the
neuron spikes much too early.</p> <title>Computer Simulation 2: Learning Spike Times</title> <p>In order to explore this learning scenario in a biologically more realistic
setting, we trained a LIF neuron with conductance based synapses exhibiting
short term facilitation and depression. The trained neuron and the neuron
<italic>μ</italic>* which produced the target spike train
<italic>S</italic>* both received inputs from 100 input neurons
emitting spikes from a constant rate Poisson process of 15 Hz. The synapses to
the trained neuron were subject to reward-modulated STDP. The weights of neuron
<italic>μ</italic>* were set to  for 0≤<italic>i</italic>&lt;50 and  for 50≤<italic>i</italic>&lt;100. In order to
simulate a non-realizable target response, neuron
<italic>μ</italic>* received 10 additional synaptic inputs (with
weights set to <italic>w<sub>max</sub></italic>/2). During the simulations we
observed a firing rate of 18.2 Hz for the trained neuron, and 25.2 Hz for the
neuron <italic>μ</italic>*. The simulations were run for 2 hours
simulated biological time.</p> <p>We performed 5 repetitions of the experiment, each time with different randomly
generated inputs and different initial weight values for the trained neuron. In
each of the 5 runs, the average synaptic weights of synapses with  and  approached their target values, as shown in ##FIG##6##Figure 7A##. In order to test
how closely the trained neuron reproduces the target spike train
<italic>S</italic>* after learning, we performed additional simulations
where the same spike input was applied to the trained neuron before and after
the learning. Then we compared the output of the trained neuron before and after
learning with the output <italic>S</italic>* of neuron
<italic>μ</italic>*. ##FIG##6##Figure 7B## shows that the trained neuron
approximates the part of <italic>S</italic>* which is accessible to it
quite well. ##FIG##6##Figure
7C–F## provide more detailed analyses of the evolution of weights
during learning. The computer simulations confirmed the theoretical prediction
that the neuron can learn well through reward-modulated STDP only if a certain
level of noise is injected into the neuron (see preceding discussion and ##SUPPL##5##Figure S6##).</p> <p>Both the theoretical results and these computer simulations demonstrate that a
neuron can learn quite well through reward-modulated STDP to respond with
specific spike patterns.</p> <title>Computer Simulation 3: Testing the Analytically Derived Conditions</title> <p>Equations 13–15 predict under which relationships between the
parameters involved the learning of particular spike responses through
reward-modulated STDP will be successful. We have tested these predictions by
selecting 6 arbitrary settings of these parameters, which are listed in ##TAB##0##Table 1##. In 4 cases (marked
by light gray shading in ##FIG##7##Figure
8##) these conditions were not met (either for the learning of weights with
target value <italic>w<sub>max</sub></italic>, or for the learning of weights
with target value 0. ##FIG##7##Figure
8## shows that the derived learning result is not achieved in exactly these
4 cases. On the other hand, the theoretically predicted weight changes (black
bar) predict in all cases the actual weight changes (gray bar) that occur for
the chosen simulation times (listed in the last column of ##TAB##0##Table 1##) remarkably well.</p> <title>Pattern Discrimination with Reward-Modulated STDP</title> <p>We examine here the question whether a neuron can learn through reward-modulated
STDP to discriminate between two spike patterns <italic>P</italic> and
<italic>N</italic> of its presynaptic neurons, by responding with more spikes to
pattern <italic>P</italic> than to pattern <italic>N</italic>. Our analysis is
based on the assumption that there exist internal rewards
<italic>d</italic>(<italic>t</italic>) that could guide such pattern
discrimination. This reward based learning architecture is biologically more
plausible than an architecture with a supervisor which provides for each input
pattern a target output and thereby directly produces the desired firing
behavior of the neuron (since the question becomes then how the supervisor has
learnt to produce the desired spike outputs).</p> <p>We consider a neuron that receives input from <italic>n</italic> presynaptic
neurons. A pattern <italic>X</italic> consists of <italic>n</italic> spike
trains, each of time length <italic>T</italic>, one for each presynaptic neuron.
There are two patterns, <italic>P</italic> and <italic>N</italic>, which are
presented in alternation to the neuron, with some reset time between
presentations. For notational simplicity, we assume that each of the
<italic>n</italic> presynaptic spike trains consists of exactly one spike.
Hence, each pattern can be defined by a list of spike times: , , where  is the time when presynaptic neuron <italic>i</italic> spikes
for pattern <italic>X</italic>∈{<italic>P</italic>,<italic>N</italic>}.
A generalization to the easier case of learning to discriminate spatio-temporal
presynaptic firing patterns (where some presynaptic neurons produce different
numbers of spikes in different patterns) is straightforward, however the main
characteristics of the learning dynamics are better accessible in this
conceptually simpler setup. It had already been shown in ##REF##17220510##[12]## that neurons
can learn through reward-modulated STDP to discriminate between different
<italic>spatial</italic> presynaptic firing patterns. But in the light of
the analysis of ##REF##17928565##[27]## it is still open whether neurons can learn
with simple forms of reward-modulated STDP, such as the one considered in this
article, to discriminate <italic>temporal</italic> presynaptic firing patterns.</p> <p>We assume that the reward signal <italic>d</italic>(<italic>t</italic>)
rewards—after some delay
<italic>d<sub>r</sub></italic>—action potentials of the trained neuron
if pattern <italic>P</italic> was presented, and punishes action potentials of
the neuron if pattern <italic>N</italic> was presented. More precisely, we
assume thatwith some reward kernel <italic>ε<sub>r</sub></italic>
and constants
<italic>α<sup>N</sup></italic>&lt;0&lt;<italic>α<sup>P</sup></italic>.
The goal of this learning task is to produce many output spikes for pattern
<italic>P</italic>, and few or no spikes for pattern <italic>N</italic>.</p> <p>The main result of our analysis is an estimate of the expected weight change of
synapse <italic>i</italic> of the trained neuron for the presentation of pattern
<italic>P</italic>, followed after a sufficiently long time
<italic>T</italic>′ by a presentation of pattern <italic>N</italic>
where 〈·〉<italic><sub>E</sub></italic>
<sub>|<italic>X</italic></sub> is the expectation over the ensemble
given that pattern <italic>X</italic> was presented. This weight change can be
estimated as (see <xref ref-type="sec" rid="s3">Methods</xref>)where <italic>ν<sup>X</sup></italic>(<italic>t</italic>)
is the postsynaptic rate at time <italic>t</italic> for pattern
<italic>X</italic>, and the constants  for
<italic>X</italic>∈{<italic>P</italic>,<italic>N</italic>} are given byAs we will see shortly, an interesting learning effect is
achieved if  is positive and  is negative. Since
<italic>f<sub>c</sub></italic>(<italic>r</italic>) is non-negative, a natural
way to achieve this is to choose a positive reward kernel
<italic>ε<sub>r</sub></italic>(<italic>r</italic>)≥0 for
<italic>r</italic>&gt;0 and
<italic>ε<sub>r</sub></italic>(<italic>r</italic>) = 0
for <italic>r</italic>&lt;0 (also,
<italic>f<sub>c</sub></italic>(<italic>r</italic>) and
<italic>ε<sub>r</sub></italic>(<italic>r</italic>) must not be
identical to zero for all <italic>r</italic>).</p> <p>We use Equation 17 to provide insight on when and how the classification of
temporal spike patterns can be learnt with reward-modulated STDP. Assume for the
moment that . We first note that it is impossible to achieve through any
synaptic plasticity rule that the time integral over the membrane potential of
the trained neuron has after training a larger value for input pattern
<italic>P</italic> than for input pattern <italic>N</italic>. The reason is that
each presynaptic neuron emits the same number of spikes in both patterns (namely
one spike). This simple fact implies that it is impossible to train a linear
Poisson neuron (with any learning method) to respond to pattern
<italic>P</italic> with more spikes than to pattern <italic>N</italic>. But
Equation 17 implies that reward-modulated STDP increases the variance of the
membrane potential for pattern <italic>P</italic>, and reduces the variance for
pattern <italic>N</italic>. This can be seen as follows. Because of the specific
form of the STDP learning curve <italic>W</italic>(<italic>r</italic>), which is
positive for (small) positive <italic>r</italic>, negative for (small) negative
<italic>r</italic>, and zero for large <italic>r</italic>,  has a potentiating effect on synapse <italic>i</italic> if the
postsynaptic rate for pattern <italic>P</italic> is larger (because of a higher
membrane potential) shortly after the presynaptic spike at this synapse
<italic>i</italic> than before that spike. This tends to further increase
the membrane potential after that spike. On the other hand, since  is negative, the same situation for pattern <italic>N</italic>
has a depressing effect on synapse <italic>i</italic>, which counteracts the
increased membrane potential after the presynaptic spike. Dually, if the
postsynaptic rate shortly after the presynaptic spike at synapse
<italic>i</italic> is lower than shortly before that spike, the effect on
synapse <italic>i</italic> is depressing for pattern <italic>P</italic>. This
leads to a further decrease of the membrane potential after that spike. In the
same situation for pattern <italic>N</italic>, the effect is potentiating, again
counteracting the variation of the membrane potential. The total effect on the
postsynaptic membrane potential is that the fluctuations for pattern
<italic>P</italic> are increased, while the membrane potential for pattern
<italic>N</italic> is flattened.</p> <p>For the LIF neuron model, and most reasonable other non-linear spiking neuron
models, as well as for biological neurons in-vivo and in-vitro ##REF##10195145##[28]##–##REF##14762148##[30]##, larger
fluctuations of the membrane potential lead to more action potentials. As a
result, reward-modulated STDP tends to increase the number of spikes for pattern
<italic>P</italic> for these neuron models, while it tends to decrease the
number of spikes for pattern <italic>N</italic>, thereby enabling a
discrimination of these purely temporal presynaptic spike patterns.</p> <title>Computer Simulation 4: Learning Pattern Classification</title> <p>We tested these theoretical predictions through computer simulations of a LIF
neuron with conductance based synapses exhibiting short-term depression and
facilitation. Both patterns, <italic>P</italic> and <italic>N</italic>, had 200
input channels, with 1 spike per channel (hence this is the extreme where
<italic>all</italic> information lies in the timing of presynaptic spikes).
The spike times were drawn from an uniform distribution over a time interval of
500 ms, which was the duration of the patterns. We performed 1000 training
trials where the patterns <italic>P</italic> and <italic>N</italic> were
presented to the neuron in alternation. To introduce exploration for this
reinforcement learning task, the neuron had injected 20% of the
Ornstein-Uhlenbeck process conductance noise (see <xref ref-type="sec" rid="s3">Methods</xref> for further details).</p> <p>The theoretical analysis predicted that the membrane potential will have after
learning a higher variance for pattern <italic>P</italic>, and a lower variance
for pattern <italic>N</italic>. When in our simulation of a LIF neuron the
firing of the neuron was switched off (by setting the firing threshold potential
too high) we could observe the membrane potential fluctuations undisturbed by
the reset mechanism after each spike (see ##FIG##8##Figure 9C and 9D##). The variance of the
membrane potential did in fact increase for pattern <italic>P</italic> from 2.49
(mV)<sup>2</sup> to 5.43 (mV)<sup>2</sup> (##FIG##8##Figure 9C##), and decrease for pattern
<italic>N</italic> (##FIG##8##Figure
9D##), from 2.34 (mV)<sup>2</sup> to 1.33 (mV)<sup>2</sup>. The
corresponding plots with the firing threshold included are given in panels E and
F, showing an increased member of spikes of the LIF neuron for pattern
<italic>P</italic>, and a decreased number of spikes for pattern
<italic>N</italic>. Furthermore, as ##FIG##8##Figure 9A and 9B## show, the increased variance of the membrane
potential for the positively reinforced pattern <italic>P</italic> led to a
stable temporal firing pattern in response to pattern <italic>P</italic>.</p> <p>We repeated the experiment 6 times, each time with different randomly generated
patterns <italic>P</italic> and <italic>N</italic>, and different random initial
synaptic weights of the neuron. The results in ##FIG##8##Figure 9G and 9H## show that the learning of
temporal pattern discrimination through reward-modulated STDP does not depend on
the temporal patterns that are chosen, nor on the initial values of synaptic
weights.</p> <title>Computer Simulation 5: Training a Readout Neuron with Reward-Modulated STDP
To Recognize Isolated Spoken Digits</title> <p>A longstanding open problem is how a biologically realistic neuron model can be
trained in a biologically plausible manner to extract information from a generic
cortical microcircuit. Previous work ##REF##12433288##[31]##–##UREF##4##[35]## has
shown that quite a bit of salient information about recent and past inputs to
the microcircuit can be extracted by a non-spiking linear readout neuron (i.e.,
a perceptron) that is trained by linear regression or margin maximization
methods. Here we examine to what extent a LIF readout neuron with conductance
based synapses (subject to biologically realistic short term synaptic
plasticity) can learn through reward-modulated STDP to extract from the response
of a simulated cortical microcircuit (consisting of 540 LIF neurons), see ##FIG##9##Figure 10A##, the information
which spoken digit (transformed into spike trains by a standard cochlea model)
is injected into the circuit. In comparison with the preceding task in
simulation 4, this task is easier because the presynaptic firing patterns that
need to be discriminated differ in temporal and spatial aspects (see ##FIG##9##Figure 10B##; ##SUPPL##9##Figure S10##
and ##SUPPL##10##S11##
show the spike trains that were injected into the circuit). But this task is on
the other hand more difficult, because the circuit response (which creates the
presynaptic firing pattern for the readout neuron) differs also significantly
for two utterances of the same digit (##FIG##9##Figure 10C##), and even for two trials for the
same utterance (##FIG##9##Figure 10D##)
because of the intrinsic noise in the circuit (which was modeled according to
##REF##11744242##[26]## to reflect in-vivo conditions during cortical
UP-states). The results shown in ##FIG##9##Figure 10E–H## demonstrate that nevertheless this learning
experiment was successful. On the other hand we were not able to achieve in this
way speaker-independent word recognition, which had been achieved in ##REF##12433288##[31]## with
a linear readout. Hence further work will be needed in order to clarify whether
biologically more realistic models for readout neurons can be trained through
reinforcement learning to reach the classification capabilities of perceptrons
that are trained through supervised learning.</p>
<title>Discussion</title> <p>We have presented in this article analytical tools which make it possible to predict
under which conditions reward-modulated STDP will achieve a given learning goal in a
network of neurons. These conditions specify relationships between parameters and
auxiliary functions (learning curves for STDP, eligibility traces, reward signals
etc.) that are involved in the specification of the reward-modulated STDP learning
rule. Although our analytical results are based on some simplifying assumptions, we
have shown that they predict quite well the outcomes of computer simulations of
quite complex models for cortical networks of neurons.</p> <p>We have applied this learning theory for reward-modulated STDP to a number of
biologically relevant learning tasks. We have shown that the biofeedback result of
Fetz and Baker ##REF##4196269##[17]## can in principle be explained on the basis of
reward-modulated STDP. The underlying credit assignment problem was extremely
difficult, since the monkey brain had no direct information about the identity of
the neuron whose firing rate was relevant for receiving rewards. This credit
assignment problem is even more difficult from the perspective of a single synapse,
and hence for the application of a local synaptic plasticity rule such as
reward-modulated STDP. However our theoretical analysis (see Equations 10 and 11)
has shown that the longterm evolution of synaptic weights depended only on the
correlation of pairs of pre- and postsynaptic spikes with the reward signal.
Therefore the firing rate of the rewarded neuron increased (for a computer
simulation of a recurrent network consisting of 4000 conductance based LIF neurons
with realistic background noise typical for in-vivo conditions, and 228954 synapses
that exhibited data-based short term synaptic plasticity) within a few minutes of
simulated biological time, like in the experimental data of ##REF##4196269##[17]##, whereas the firing rates
of the other neurons remained invariant (see ##FIG##3##Figure 4B##). We were also able to model
differential reinforcement of two neurons in this way (##FIG##1##Figure 2##). These computer simulations
demonstrated a remarkable stability of the network dynamics (see ##FIG##1##Figures 2A##, ##FIG##3##4A##, and ##FIG##4##5##) in spite of the fact that all excitatory
synapses were continuously subjected to reward-modulated STDP. In particular, the
circuit remained in the asynchronous irregular firing regime, that resembles
spontaneous firing activity in the cortex ##REF##10676963##[9]##. Other STDP-rules
(without reward modulation) that maintain this firing regime have previously been
exhibited in ##REF##17444756##[22]##. It was also reported in ##REF##4196269##[17]##, and further examined in
##REF##810359##[46]##,
that bursts of the reinforced neurons were often accompanied by activations of
specific muscles in the biofeedback experiment by Fetz and Baker. But the
relationship between bursts of the recorded neurons in precentral motor cortex and
muscle activations was reported to be quite complex and often dropped out after
continued reinforcement of the neuron alone. Furthermore in ##REF##810359##[46]## it was shown that all
neurons tested in that study could be dissociated from their correlated muscle
activity by differentially reinforcing simultaneous suppression of EMG activity.
These results suggest that the solution of the credit assignment problem by the
monkeys (to stronger activate that neuron out of billions of neurons in their
precentral gyrus that was reinforced) may have been supported by large scale
exploration strategies that were associated with muscle activations. But the
previously mentioned results on differential reinforcements of two nearby neurons
suggest that this large scale exploration strategy had to be complemented by
exploration on a finer spatial scale that is difficult to explain on the basis of
muscle activations (see ##REF##17234689##[19]## for a detailed discussion).</p> <p>Whereas this learning task focused on firing rates, we have also shown (see ##FIG##6##Figure 7##) that neurons can learn
via reward-modulated STDP to respond to inputs with particular spike trains, i.e.,
particular temporal output patterns. It has been pointed out in ##REF##17928565##[27]## that
this is a particularly difficult learning task for reward-modulated STDP, and it was
shown there that it can be accomplished with a modified STDP rule and more complex
reward prediction signals without delays. We have complemented the results of ##REF##17928565##[27]## by
deriving specific conditions (Equations 13–15) under which this learning
task can be solved by the standard version of reward-modulated STDP. Extensive
computer simulations have shown that these analytically derived conditions for a
simpler neuron model predict also for a LIF neuron with conductance based synapses
whether it is able to solve this learning task. ##FIG##7##Figure 8## shows that this learning theory for
reward-modulated STDP is also able to predict quite well <italic>how fast</italic> a
neuron can learn to produce a desired temporal output pattern. An interesting aspect
of ##REF##17928565##[27]##
is that there also the utility of third signals that provide information about
changes in the expectation of reward was explored. We have considered in this
article only learning scenarios where reward prediction is not possible. A logical
next step will be to extend our learning theory for reward-modulated STDP to
scenarios from classical reinforcement learning theory that include reward
prediction.</p> <p>We have also addressed the question to what extent neurons can learn via
reward-modulated STDP to respond with different firing rates to different
spatio-temporal presynaptic firing patterns. It had already been shown in ##REF##17220510##[12]##
that this learning rule enables neurons to classify spatial firing patterns. We have
complemented this work by deriving an analytic expression for the expected weight
change in this learning scenario (see Equation 17), which clarifies to what extent a
neuron can learn by reward-modulated STDP to distinguish differences in the temporal
structure of presynaptic firing patterns. This theoretical analysis showed that in
the extreme case, where all incoming information is encoded in the relative timing
of presynaptic spikes, reward-modulated STDP is not able to produce a higher average
membrane potential for selected presynaptic firing patterns, even if that would be
rewarded. But it is able to increase the variance of the membrane potential, and
thereby also the number of spikes of any neuron model that has (unlike the simple
linear Poisson neuron) a firing threshold. The simulation results in ##FIG##8##Figure 9## confirm that in this way
a LIF neuron can learn with the standard version of reward-modulated STDP to
discriminate even purely temporal presynaptic firing patterns, by producing more
spikes in response to one of these patterns.</p> <p>A surprising feature is, that although the neuron was rewarded here only for
responding with a higher firing rate to one presynaptic firing pattern
<italic>P</italic>, it automatically started to respond to this pattern
<italic>P</italic> with a specific temporal spike pattern, that advanced in time
during training (see ##FIG##8##Figure 9A##).</p> <p>Finally, we have shown that a spiking neuron can be trained by reward-modulated STDP
to read out information from a simulated cortical microcircuit (see ##FIG##9##Figure 10##). This is insofar of
interest, as previous work ##REF##12433288##[31]##,##UREF##3##[34]##,##REF##16481565##[47]## had shown that models of generic cortical
microcircuits have inherent capabilities to serve as preprocessors for such readout
neurons, by combining in diverse linear and nonlinear ways information that was
contained in different time segments of spike inputs to the circuit
(“liquid computing model”). The classification of spoken words
(that were first transformed into spike trains) had been introduced as a common
benchmark task for the evaluation of different approaches towards computing with
spiking neurons ##REF##12433288##[31]##–##REF##15483600##[33]##,##UREF##8##[45]##,##REF##11158631##[48]##. But
so far all approaches that were based on learning (rather than on clever
constructions) had to rely on supervised training of a simple linear readout. This
gave rise to the question whether also biologically more realistic models for
readout neurons can be trained through a biologically more plausible learning
scenario to classify spoken words. The results of ##FIG##9##Figure 10## may be interpreted as a tentative
positive answer to this question. We have demonstrated that LIF neurons with
conductance based synapses (that are subject to biologically realistic short term
plasticity) can learn without a supervisor through reward-modulated STDP to classify
spoken digits. In contrast to the result of ##FIG##8##Figure 9##, the output code that emerged here was a
rate code. This can be explained through the significant in-class variance of
circuit responses to different utterances of the same word (see ##FIG##9##Figure 10C and 10D##). Although the LIF neuron
learnt here without a supervisor to respond with different firing rates to
utterances of different words by the same speaker (whereas the rate output was very
similar for both words at the beginning of learning, see ##FIG##9##Figure 10E##), the classification capability of
these neurons has not yet reached the level of linear readouts that are trained by a
supervisor (for example, speaker independent word classification could not yet be
achieved in this way). Further work is needed to test whether the classification
capability of LIF readout neurons can be improved through additional preprocessing
in the cortical microcircuit model, through a suitable variation of the
reward-modulated STDP rule, or through a different learning scenario (mimicking for
example preceding developmental learning that also modifies the presynaptic
circuit).</p> <p>The new learning theory for reward-modulated STDP will also be useful for biological
experiments that aim at the clarification of details of the biological
implementation of synaptic plasticity in different parts of the brain, since it
allows to make predictions which types and time courses of signals would be optimal
for a particular range of learning tasks. For each of the previously discussed
learning tasks, the theoretical analysis provided conditions on the structure of the
reward signal <italic>d</italic>(<italic>t</italic>) which guaranteed successful
learning. For example, in the biofeedback learning scenario (##FIG##3##Figure 4##), every action potential of the
reinforced neuron led—after some delay—to a change of the reward
signal <italic>d</italic>(<italic>t</italic>). The shape of this change was defined
by the reward kernel <italic>ε</italic>(<italic>r</italic>). Our analysis
revealed that this reward kernel can be chosen rather arbitrarily as long as the
integral over the kernel is zero, and the integral over the product of the kernel
and the eligibility function is positive. For another learning scenario, where the
goal was that the output spike train  of some neuron <italic>j</italic> approximates the spike timings
of some target spike train <italic>S</italic>* (##FIG##6##Figure 7##), the reward signal has to depend on
both,  and <italic>S</italic>*. The dependence of the reward
signal on these spike timings was defined by a reward kernel
<italic>κ</italic>(<italic>r</italic>). Our analysis showed that the
reward kernel has to be chosen for this task so that the synapses receive positive
rewards if the postsynaptic neuron fires close to the time of a spike in the target
spike train <italic>S</italic>* or somewhat later, and negative rewards when
an output spike occurs in the order of ten milliseconds too early. In the pattern
discrimination task of ##FIG##8##Figure 9##
each postsynaptic action potential was followed—after some
delay—by a change of the reward signal which depended on the pattern
presented. Our theoretical analysis predicted that this learning task can be solved
if the integrals  and  defined by Equation 18 are such that  and . Again, this constraints are fulfilled for a large class of reward
kernels, and a natural choice is to use a non-negative reward kernel
<italic>ε<sub>r</sub></italic>. There are currently no data
available on the shape of reward kernels in biological neural systems. The previous
sketched theoretical analysis makes specific prediction for the shape of reward
kernels (depending on the type of learning task in which a biological neural system
is involved) which can potentially be tested through biological experiments.</p> <p>An interesting general aspect of the learning theory that we have presented in this
article is that it requires substantial trial-to-trial variability in the neural
circuit, which is often viewed as “noise” of imperfect
biological implementations of theoretically ideal circuits of neurons. This learning
theory for reward-modulated STDP suggests that the main functional role of noise is
to maintain a suitable level of spontaneous firing (since if a neuron does not fire,
it cannot find out whether this will be rewarded), which should vary from trial to
trial in order to explore which firing patterns are rewarded (It had been shown in
##REF##12433288##[31]##,##UREF##3##[34]##,##REF##16481565##[47]## that such highly variable circuit activity is
compatible with a stable performance of linear readouts). On the other hand if a
neuron fires primarily on the basis of a noise current that is directly injected
into that neuron, and not on the basis of presynaptic activity, then STDP does not
have the required effect on the synaptic connections to this neuron (see ##SUPPL##5##Figure S6##).
This perspective opens the door for subsequent studies that compare for concrete
biological learning tasks the theoretically derived optimal amount and distribution
of trial-to-trial variability with corresponding experimental data.</p> <title>Related Work</title> <p>The theoretical analysis of this model is directly applicable to the learning
rule considered in ##REF##17220510##[12]##. There, the network behavior of
reward-modulated STDP was also studied some situations different from the ones
in this article. The computer simulations of ##REF##17220510##[12]## operate
apparently in a different dynamic regime, where LTD dominates LTP in the
STDP-rule, and most weights (except those that are actively increased through
reward-modulated STDP) have values close to 0 (see Figure 1b and 1d in ##REF##17220510##[12]##, and compare
with ##FIG##4##Figure 5## in this
article). This setup is likely to require for successful learning a larger
dominance of pre-before-post over post-before-pre pairs than the one shown in
##FIG##3##Figure 4E##. Furthermore,
whereas a very low spontaneous firing rate of 1 Hz was required in ##REF##17220510##[12]##, computer simulation 1 shows that reinforcement
learning is also feasible at spontaneous firing rates which correspond to those
reported in ##REF##4196269##[17]## (the preceding theoretical analysis had
already suggested that the success of the model does not depend on particularly
low firing rates). The articles ##REF##17571943##[15]## and ##UREF##0##[13]## investigate
variations of reward-modulated STDP rules that do not employ learning curves for
STDP that are based on experimental data, but modified curves that arise in the
context of a very interesting top-down theoretical approach (distributed
reinforcement learning ##UREF##1##[14]##). The authors of ##REF##16764506##[16]## arrive at similar
learning rules in a supervised scenario which can be reinterpreted in the
context of reinforcement learning. We expect that a similar theory as we have
presented in this article for the more commonly discussed version of STDP can
also be applied to their modified STDP rules, thereby making it possible to
predict under which conditions their learning rules will succeed. Another reward
based learning rule for spiking neurons was recently presented in ##REF##16907616##[49]##.
This rule exploits correlations of a reward signal with noisy perturbations of
the neuronal membrane conductance in order to optimize some objective function.
One crucial assumption of this approach is that the synaptic plasticity
mechanism “knows” which contributions to the membrane
potential arise from synaptic inputs, and which contributions are due to
internal noise. Such explicit knowledge of the noise signal is not needed in the
reward-modulated STDP rule of ##REF##17220510##[12]##, which we have
considered in this article. The price one has to pay for this potential gain in
biological realism is a reduced generality of the learning capabilities. While
the learning rule in ##REF##16907616##[49]## approximates gradient ascent on the objective
function, this cannot be stated for reward-modulated STDP at present.
Timing-based pattern discrimination with a spiking neuron, as discussed in the
section “Pattern discrimination with reward-modulated STDP”
of this article, was recently tackled in ##REF##16474393##[50]##. The authors proposed
the tempotron learning rule, which increases the peak membrane voltage for one
class of input patterns (if no spike occurred in response to the input pattern)
while decreasing the peak membrane voltage for another class of input patterns
(if a spike occurred in response to the pattern). The main difference between
this learning rule and reward-modulated STDP is that the tempotron learning rule
is sensitive to the peak membrane voltage, whereas reward-modulated STDP is
sensitive to local fluctuations of the membrane voltage. Since the time of the
maximal membrane voltage has to be determined for each pattern by the synaptic
plasticity mechanism, the basic tempotron rule is perhaps not biologically
realistic. Therefore, an approximate and potentially biologically more realistic
learning rule was proposed in ##REF##16474393##[50]##, where plasticity following error trials is
induced at synapse <italic>i</italic> only if the voltage within the
postsynaptic integration time after their activation exceeds a plasticity
threshold <italic>κ</italic>. One potential problem of this rule is the
plasticity threshold <italic>κ</italic>, since a good choice of this
parameter strongly depends on the mean membrane voltage after input spikes. This
problem is circumvented by reward-modulated STDP, which considers instead the
local change in the membrane voltage. Further work is needed to compare the
advantages and disadvantages of these different approaches.</p> <title>Conclusion</title> <p>Reward-modulated STDP is a very promising candidate for a synaptic plasticity
rule that is able to orchestrate local synaptic modifications in such a way that
particular functional properties of larger networks of neurons can be achieved
and maintained (we refer to ##REF##17220510##[12]## and ##REF##17928565##[27]## for discussion of
potential biological implementations of this plasticity rule). We have provided
in this article analytical tools which make it possible to evaluate this rule
and variations of this rule not just through computer simulations, but through
theoretical analysis. In particular we have shown that successful learning is
only possible if certain relationships hold between the parameters that are
involved. Some of these predicted relationships can be tested through biological
experiments.</p> <p>Provided that these relationships are satisfied, reward-modulated STDP turns out
to be a powerful rule that can achieve self-organization of synaptic weights in
large recurrent networks of neurons. In particular, it enables us to explain
seemingly inexplicable experimental data on biofeedback in monkeys. In addition
reward-modulated STDP enables neurons to distinguish complex firing patterns of
presynaptic neurons, even for data-based standard forms of STDP, and without the
need for a supervisor that tells the neuron when it should spike. Furthermore
reward-modulated STDP requires substantial spontaneous activity and
trial-to-trial variability in order to support successful learning, thereby
providing a functional explanation for these ubiquitous features of cortical
networks of neurons. In fact, not only spontaneous activity but also STDP itself
may be seen in this context as a mechanism that supports the exploration of
different firing chains within a recurrent network, until a solution is found
that is rewarded because it supports a successful computational function of the
network.</p>
<title>Conclusion</title> <p>Reward-modulated STDP is a very promising candidate for a synaptic plasticity
rule that is able to orchestrate local synaptic modifications in such a way that
particular functional properties of larger networks of neurons can be achieved
and maintained (we refer to ##REF##17220510##[12]## and ##REF##17928565##[27]## for discussion of
potential biological implementations of this plasticity rule). We have provided
in this article analytical tools which make it possible to evaluate this rule
and variations of this rule not just through computer simulations, but through
theoretical analysis. In particular we have shown that successful learning is
only possible if certain relationships hold between the parameters that are
involved. Some of these predicted relationships can be tested through biological
experiments.</p> <p>Provided that these relationships are satisfied, reward-modulated STDP turns out
to be a powerful rule that can achieve self-organization of synaptic weights in
large recurrent networks of neurons. In particular, it enables us to explain
seemingly inexplicable experimental data on biofeedback in monkeys. In addition
reward-modulated STDP enables neurons to distinguish complex firing patterns of
presynaptic neurons, even for data-based standard forms of STDP, and without the
need for a supervisor that tells the neuron when it should spike. Furthermore
reward-modulated STDP requires substantial spontaneous activity and
trial-to-trial variability in order to support successful learning, thereby
providing a functional explanation for these ubiquitous features of cortical
networks of neurons. In fact, not only spontaneous activity but also STDP itself
may be seen in this context as a mechanism that supports the exploration of
different firing chains within a recurrent network, until a solution is found
that is rewarded because it supports a successful computational function of the
network.</p>
<p>Conceived and designed the experiments: RL DP WM. Wrote the paper: RL DP
WM.</p> <p>Reward-modulated spike-timing-dependent plasticity (STDP) has recently emerged as
a candidate for a learning rule that could explain how behaviorally relevant
adaptive changes in complex networks of spiking neurons could be achieved in a
self-organizing manner through local synaptic plasticity. However, the
capabilities and limitations of this learning rule could so far only be tested
through computer simulations. This article provides tools for an analytic
treatment of reward-modulated STDP, which allows us to predict under which
conditions reward-modulated STDP will achieve a desired learning effect. These
analytical results imply that neurons can learn through reward-modulated STDP to
classify not only spatial but also temporal firing patterns of presynaptic
neurons. They also can learn to respond to specific presynaptic firing patterns
with particular spike patterns. Finally, the resulting learning theory predicts
that even difficult credit-assignment problems, where it is very hard to tell
which synaptic weights should be modified in order to increase the global reward
for the system, can be solved in a self-organizing manner through
reward-modulated STDP. This yields an explanation for a fundamental experimental
result on biofeedback in monkeys by Fetz and Baker. In this experiment monkeys
were rewarded for increasing the firing rate of a particular neuron in the
cortex and were able to solve this extremely difficult credit assignment
problem. Our model for this experiment relies on a combination of
reward-modulated STDP with variable spontaneous firing activity. Hence it also
provides a possible functional explanation for trial-to-trial variability, which
is characteristic for cortical networks of neurons but has no analogue in
currently existing artificial computing systems. In addition our model
demonstrates that reward-modulated STDP can be applied to all synapses in a
large recurrent neural network without endangering the stability of the network
dynamics.</p> <title>Author Summary</title> <p>A major open problem in computational neuroscience is to explain how learning,
i.e., behaviorally relevant modifications in the central nervous system, can be
explained on the basis of experimental data on synaptic plasticity.
Spike-timing-dependent plasticity (STDP) is a rule for changes in the strength
of an individual synapse that is supported by experimental data from a variety
of species. However, it is not clear how this synaptic plasticity rule can
produce meaningful modifications in networks of neurons. Only if one takes into
account that consolidation of synaptic plasticity requires a third signal, such
as changes in the concentration of a neuromodulator (that might, for example, be
related to rewards or expected rewards), then meaningful changes in the
structure of networks of neurons may occur. We provide in this article an
analytical foundation for such reward-modulated versions of STDP that predicts
when this type of synaptic plasticity can produce functionally relevant changes
in networks of neurons. In particular we show that seemingly inexplicable
experimental data on biofeedback, where a monkey learnt to increase the firing
rate of an arbitrarily chosen neuron in the motor cortex, can be explained on
the basis of this new learning theory.</p>
<title>Supporting Information</title>
<p>We would like to thank Markus Diesmann, Eberhard Fetz, Razvan Florian, Yves Fregnac,
Wulfram Gerstner, Nikos Logothetis, Abigail Morrison, Matthias Munk, Gordon Pipa,
and Dan Shulz for helpful discussions. In addition we would like to thank Malcolm
Slaney for providing a MATLAB implementation of the cochlea model from ##UREF##6##[43]##, as well
as Benjamin Schrauwen, David Verstraeten, Michiel D'Haene, and Stefan
Klampfl for additional code that we used in our speech classification task (computer
simulation 5).</p>
<fig id="pcbi-1000180-g001" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g001</object-id><label>Figure 1</label><caption><title>Scheme of reward-modulated STDP according to Equations 1–4.</title><p>(A) Eligibility function <italic>f<sub>c</sub></italic>(<italic>t</italic>),
which scales the contribution of a pre/post spike pair (with the second
spike at time 0) to the eligibility trace
<italic>c</italic>(<italic>t</italic>) at time <italic>t</italic>. (B)
Contribution of a pre-before-post spike pair (in red) and a post-before-pre
spike pair (in green) to the eligibility trace
<italic>c</italic>(<italic>t</italic>) (in black), which is the sum of the
red and green curves. According to Equation 1 the change of the synaptic
weight <italic>w</italic> is proportional to the product of
<italic>c</italic>(<italic>t</italic>) with a reward signal
<italic>d</italic>(<italic>t</italic>).</p></caption></fig> <fig id="pcbi-1000180-g002" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g002</object-id><label>Figure 2</label><caption><title>Differential reinforcement of two neurons (within a simulated network
of 4000 neurons, the two rewarded neurons are denoted as A and B),
corresponding to the experimental results shown in Figure 9 of ##REF##4196269##[17]## and Figure 1 of ##REF##17234689##[19]##.</title><p>(A) The spike response of 100 randomly chosen neurons at the beginning of
the simulation (20 sec–23 sec, left plot), and at the middle
of simulation just before the switching of the reward policy (597
sec–600 sec, right plot). The firing times of the first
reinforced neuron A are marked by blue crosses and those of the second
reinforced neuron B are marked by green crosses. (B) The dashed vertical
line marks the switch of the reinforcements at
<italic>t</italic> = 10 min. The firing
rate of neuron A (blue line) increases while it is positively reinforced
in the first half of the simulation and decreases in the second half
when its spiking is negatively reinforced. The firing rate of the neuron
B (green line) decreases during the negative reinforcement in the first
half and increases during the positive reinforcement in the second half
of the simulation. The average firing rate of 20 other randomly chosen
neurons (dashed line) remains unchanged. (C) Evolution of the average
weight of excitatory synapses to the rewarded neurons A and B (blue and
green lines, respectively), and of the average weight of 1744 randomly
chosen excitatory synapses to other neurons in the circuit (dashed
line).</p></caption></fig> <fig id="pcbi-1000180-g003" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g003</object-id><label>Figure 3</label><caption><title>Setup of the model for the experiment by Fetz and Baker ##REF##4196269##[17]##.</title><p>(A) Schema of the model: The activity of a single neuron in the circuit
determines the amount of reward delivered to all synapses between
excitatory neurons in the circuit. (B) The reward signal
<italic>d</italic>(<italic>t</italic>) in response to a spike train
(shown at the top) of the arbitrarily selected neuron (which was
selected from a recurrently connected circuit consisting of 4000
neurons). The level of the reward signal
<italic>d</italic>(<italic>t</italic>) follows the firing rate of the
spike train. (C) The eligibility function
<italic>f<sub>c</sub></italic>(<italic>s</italic>) (black curve,
left axis), the reward kernel
<italic>ε<sub>r</sub></italic>(<italic>s</italic>) delayed
by 200 ms (red curve, right axis), and the product of these two
functions (blue curve, right axis) as used in our computer experiment.
The integral of
<italic>f<sub>c</sub></italic>(<italic>s</italic>+<italic>d<sub>r</sub></italic>)<italic>ε<sub>r</sub></italic>(<italic>s</italic>)
is positive, as required according to Equation 10 in order to achieve a
positive learning rate for the synapses to the selected neuron.</p></caption></fig> <fig id="pcbi-1000180-g004" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g004</object-id><label>Figure 4</label><caption><title>Simulation of the experiment by Fetz and Baker ##REF##4196269##[17]## for the case
where an arbitrarily selected neuron triggers global rewards when it
increases its firing rate.</title><p>(A) Spike response of 100 randomly chosen neurons within the recurrent
network of 4000 neurons at the beginning of the simulation (20
sec–23 sec, left plot), and at the end of the simulation (the
last 3 seconds, right plot). The firing times of the reinforced neuron
are marked by blue crosses. (B) The firing rate of the positively
rewarded neuron (blue line) increases, while the average firing rate of
20 other randomly chosen neurons (dashed line) remains unchanged. (C)
Evolution of the average weight of excitatory synapses to the reinforced
neuron (blue line), and of the average weight of 1663 randomly chosen
excitatory synapses to other neurons in the circuit (dashed line). (D)
Spike trains of the reinforced neuron before and after learning. (E)
Histogram of the time-differences between presynaptic and postsynaptic
spikes (bin size 0.5 ms), averaged over all excitatory synapses to the
reinforced neuron. The black curve represents the histogram values for
positive time differences (when the presynaptic spike precedes the
postsynaptic spike), and the red curve represents the histogram for
negative time differences.</p></caption></fig> <fig id="pcbi-1000180-g005" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g005</object-id><label>Figure 5</label><caption><title>Evolution of the dynamics of a recurrent network of 4000 LIF neurons
during application of reward-modulated STDP.</title><p>(A) Distribution of the synaptic weights of excitatory synapses to 50
randomly chosen non-reinforced neurons, plotted for 4 different periods
of simulated biological time during the simulation. The weights are
averaged over 10 samples within these periods. The colors of the curves
and the corresponding intervals are as follows: red (300–360
sec), green (600–660 sec), blue (900–960 sec),
magenta (1140–1200 sec). (B) The distribution of average
firing rates of the non-reinforced excitatory neurons in the circuit,
plotted for the same time periods as in (A). The colors of the curves
are the same as in (A). The distribution of the firing rates of the
neurons in the circuit remains unchanged during the simulation, which
covers 20 minutes of biological time. (C) Cross-correlogram of the
spiking activity in the circuit, averaged over 200 pairs of
non-reinforced neurons and over 60 s, with a bin size of 0.2 ms, for the
period between 300 and 360 seconds of simulated biological time. It is
calculated as the cross-covariance divided by the square root of the
product of variances. (D) As in (C), but between seconds 1140 and 1200.
(Separate plots of (B), (C), and (D) for two types of excitatory neurons
that received different amounts of noise currents are given in ##SUPPL##0##Figure
S1## and ##SUPPL##1##Figure S2##.)</p></caption></fig> <fig id="pcbi-1000180-g006" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g006</object-id><label>Figure 6</label><caption><title>Setup for reinforcement learning of spike times.</title><p>(A) Architecture. The trained neuron receives <italic>n</italic> input
spike trains. The neuron <italic>μ</italic>* receives
the same inputs plus additional inputs not accessible to the trained
neuron. The reward is determined by the timing differences between the
action potentials of the trained neuron and the neuron
<italic>μ</italic>*. (B) A reward kernel with optimal
offset from the origin of
<italic>t<sub>κ</sub></italic> = −6.6
ms. The optimal offset for this kernel was calculated with respect to
the parameters from computer simulation 1 in ##TAB##0##Table 1##. Reward is positive if the
neuron spikes around the target spike or somewhat later, and negative if
the neuron spikes much too early.</p></caption></fig> <fig id="pcbi-1000180-g007" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g007</object-id><label>Figure 7</label><caption><title>Results for reinforcement learning of exact spike times through
reward-modulated STDP.</title><p>(A) Synaptic weight changes of the trained LIF neuron, for 5 different
runs of the experiment. The curves show the average of the synaptic
weights that should converge to  (dashed lines), and the average of the synaptic
weights that should converge to  (solid lines) with different colors for each
simulation run. (B) Comparison of the output of the trained neuron
before (top trace) and after learning (bottom trace). The same input
spike trains and the same noise inputs were used before and after
training for 2 hours. The second trace from above shows those spike
times <italic>S</italic>* which are rewarded, the third trace
shows the realizable part of <italic>S</italic>* (i.e. those
spikes which the trained neuron could potentially learn to reproduce,
since the neuron <italic>μ</italic>* produces them
without its 10 extra spike inputs). The close match between the third
and fourth trace shows that the trained neuron performs very well. (C)
Evolution of the spike correlation between the spike train of the
trained neuron and the realizable part of the target spike train
<italic>S</italic>*. (D) The angle between the weight vector
w of the trained neuron and the weight vector w* of the neuron
<italic>μ</italic>* during the simulation, in
radians. (E) Synaptic weights at the beginning of the simulation are
marked with ×, and at the end of the simulation with
•, for each plastic synapse of the trained neuron. (F)
Evolution of the synaptic weights
<italic>w</italic>/<italic>w<sub>max</sub></italic> during the
simulation (we had chosen  for <italic>i</italic>&lt;50,  for <italic>i</italic>≥50).</p></caption></fig> <fig id="pcbi-1000180-g008" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g008</object-id><label>Figure 8</label><caption><title>Test of the validity of the analytically derived conditions
13–15 on the relationship between parameters for successful
learning with reward-modulated STDP.</title><p>Predicted average weight changes (black bars) calculated from Equation 22
match in sign and magnitude the actual average weight changes (gray
bars) in computer simulations, for 6 different experiments with
different parameter settings (see ##TAB##0##Table 1##). (A) Weight changes for
synapses with . (B) Weight changes for synapses with . Four cases where constraints 13–15 are not
fulfilled are shaded in light gray. In all of these four cases the
weights move into the opposite direction, i.e., a direction that
decreases rewards.</p></caption></fig> <fig id="pcbi-1000180-g009" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g009</object-id><label>Figure 9</label><caption><title>Training a LIF neuron to classify purely temporal presynaptic firing
patterns: a positive reward is given for firing of the neuron in
response to a temporal presynaptic firing pattern <italic>P</italic>,
and a negative reward for firing in response to another temporal pattern
<italic>N</italic>.</title><p>(A) The spike response of the neuron for individual trials, during 500
training trials when pattern <italic>P</italic> is presented. Only the
spikes from every 4-th trial are plotted. (B) As in (A), but in response
to pattern <italic>N</italic>. (C) The membrane potential
<italic>V<sub>m</sub></italic>(<italic>t</italic>) of the neuron
during a trial where pattern <italic>P</italic> is presented, before
(blue curve) and after training (red curve), with the firing threshold
removed. The variance of the membrane potential increases during
learning, as predicted by the theory. (D) As in (C), but for pattern
<italic>N</italic>. The variance of the membrane potential for
pattern <italic>N</italic> decreases during learning, as predicted by
the theory. (E) The membrane potential
<italic>V<sub>m</sub></italic>(<italic>t</italic>) of the neuron
(including action potentials) during a trial where pattern
<italic>P</italic> is presented before (blue curve) and after training
(red curve). The number of spikes increases. (F) As in (E), but for
trials where pattern <italic>N</italic> is given as input. The number of
spikes decreases. (G) Average number of output spikes per trial before
learning, in response to pattern <italic>P</italic> (gray bars) and
pattern <italic>N</italic> (black bars), for 6 experiments with
different randomly generated patterns <italic>P</italic> and
<italic>N</italic>, and different random initial synaptic weights of the
neuron. (H) As in (G), for the same experiments, but after learning. The
average number of spikes per trial increases after training for pattern
<italic>P</italic>, and decreases for pattern
<italic>N</italic>.</p></caption></fig> <fig id="pcbi-1000180-g010" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.g010</object-id><label>Figure 10</label><caption><title>A LIF neuron is trained through reward-modulated STDP to discriminate
as a “readout neuron” responses of generic cortical
microcircuits to utterances of different spoken digits.</title><p>(A) Circuit response to an utterance of digit “one”
(spike trains of 200 out of 540 neurons in the circuit are shown). The
response within the time period from 100 to 200 ms (marked in gray) is
used as a reference in the subsequent 3 panels. (B) The circuit response
from (A) (black) for the period between 100 and 200 ms, and the circuit
response to an utterance of digit “two” (red). (C)
The circuit spike response from (A) (black) and a circuit response for
another utterance of digit “one” (red), also shown
for the period between 100 and 200 ms. (D) The circuit spike response
from (A) (black), and another circuit response to the same utterance in
another trial (red). The responses differ due to the presence of noise
in the circuit. (E) Spike response of the LIF readout neuron for
different trials during learning, for trials where utterances of digit
“two” (left plot) and digit
“one” (right plot) are presented as circuit inputs.
The spikes from each 4th trial are plotted. (F) Average number of spikes
in the response of the readout during training, in response to digit
“one” (blue) and digit “two”
(green). The number of spikes were averaged over 40 trials. (G) The
membrane potential <italic>V<sub>m</sub></italic>(<italic>t</italic>) of
the neuron during a trial where an input pattern corresponding to an
utterance of digit “two” is presented, before (blue
curve) and after training (red curve), with the firing threshold
removed. (H) As in (G), but for an input pattern corresponding to an
utterance of digit “one”. The variance of the
membrane potential increases during learning for utterances of the
rewarded digit, and decreases for the non-rewarded digit.</p></caption></fig>
<table-wrap id="pcbi-1000180-t001" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.t001</object-id><label>Table 1</label><caption><title>Parameter values used for computer simulation 3 (see ##FIG##7##Figure 8##).</title></caption><alternatives><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/></colgroup><thead><tr><td align="left" rowspan="1" colspan="1">Ex.</td><td align="left" rowspan="1" colspan="1">
<italic>τ<sub>ε</sub></italic>
[ms]</td><td align="left" rowspan="1" colspan="1">
<italic>w<sub>max</sub></italic>
</td><td align="left" rowspan="1" colspan="1">
<italic>υ<sup>post</sup><sub>min</sub></italic>
[Hz]</td><td align="left" rowspan="1" colspan="1">
<italic>A<sub>+</sub></italic>
10<sup>6</sup>
</td><td align="left" rowspan="1" colspan="1">
<italic>A<sub>−</sub></italic>/<italic>A<sub>+</sub></italic>
</td><td align="left" rowspan="1" colspan="1">
<italic>τ<sub>+</sub></italic>
[ms]</td><td align="left" rowspan="1" colspan="1">
<italic>A<sup>κ</sup><sub>+</sub></italic>,
<italic>A<sup>κ</sup><sub>−</sub></italic>
</td><td align="left" rowspan="1" colspan="1">
<italic>τ<sup>κ</sup></italic>
<sub>2</sub>
[ms]</td><td align="left" rowspan="1" colspan="1">
<italic>t<sub>sim</sub></italic>
[h]</td></tr></thead><tbody><tr><td align="left" rowspan="1" colspan="1">1</td><td align="left" rowspan="1" colspan="1">10</td><td align="left" rowspan="1" colspan="1">0.012</td><td align="left" rowspan="1" colspan="1">10</td><td align="left" rowspan="1" colspan="1">16.62</td><td align="left" rowspan="1" colspan="1">1.05</td><td align="left" rowspan="1" colspan="1">20</td><td align="left" rowspan="1" colspan="1">3.34, −3.12</td><td align="left" rowspan="1" colspan="1">20</td><td align="left" rowspan="1" colspan="1">5</td></tr><tr><td align="left" rowspan="1" colspan="1">2</td><td align="left" rowspan="1" colspan="1">7</td><td align="left" rowspan="1" colspan="1">0.020</td><td align="left" rowspan="1" colspan="1">5</td><td align="left" rowspan="1" colspan="1">11.08</td><td align="left" rowspan="1" colspan="1">1.02</td><td align="left" rowspan="1" colspan="1">15</td><td align="left" rowspan="1" colspan="1">4.58, −4.17</td><td align="left" rowspan="1" colspan="1">16</td><td align="left" rowspan="1" colspan="1">10</td></tr><tr><td align="left" rowspan="1" colspan="1">3</td><td align="left" rowspan="1" colspan="1">20</td><td align="left" rowspan="1" colspan="1">0.010</td><td align="left" rowspan="1" colspan="1">6</td><td align="left" rowspan="1" colspan="1">5.54</td><td align="left" rowspan="1" colspan="1">1.10</td><td align="left" rowspan="1" colspan="1">25</td><td align="left" rowspan="1" colspan="1">1.50, −1.39</td><td align="left" rowspan="1" colspan="1">40</td><td align="left" rowspan="1" colspan="1">19</td></tr><tr><td align="left" rowspan="1" colspan="1">4</td><td align="left" rowspan="1" colspan="1">7</td><td align="left" rowspan="1" colspan="1">0.020</td><td align="left" rowspan="1" colspan="1">5</td><td align="left" rowspan="1" colspan="1">11.08</td><td align="left" rowspan="1" colspan="1">1.07</td><td align="left" rowspan="1" colspan="1">25</td><td align="left" rowspan="1" colspan="1">4.67, −4.17</td><td align="left" rowspan="1" colspan="1">16</td><td align="left" rowspan="1" colspan="1">13</td></tr><tr><td align="left" rowspan="1" colspan="1">5</td><td align="left" rowspan="1" colspan="1">10</td><td align="left" rowspan="1" colspan="1">0.015</td><td align="left" rowspan="1" colspan="1">6</td><td align="left" rowspan="1" colspan="1">20.77</td><td align="left" rowspan="1" colspan="1">1.10</td><td align="left" rowspan="1" colspan="1">25</td><td align="left" rowspan="1" colspan="1">3.75, −3.12</td><td align="left" rowspan="1" colspan="1">20</td><td align="left" rowspan="1" colspan="1">2</td></tr><tr><td align="left" rowspan="1" colspan="1">6</td><td align="left" rowspan="1" colspan="1">25</td><td align="left" rowspan="1" colspan="1">0.005</td><td align="left" rowspan="1" colspan="1">3</td><td align="left" rowspan="1" colspan="1">13.85</td><td align="left" rowspan="1" colspan="1">1.01</td><td align="left" rowspan="1" colspan="1">25</td><td align="left" rowspan="1" colspan="1">3.34, −3.12</td><td align="left" rowspan="1" colspan="1">20</td><td align="left" rowspan="1" colspan="1">18</td></tr></tbody></table></alternatives></table-wrap> <table-wrap id="pcbi-1000180-t002" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.t002</object-id><label>Table 2</label><caption><title>Mean values of the U, D, and F parameters in the model from ##REF##9560274##[37]## for the short-term dynamics of synapses,
depending on the type of the presynaptic and postsynaptic neuron
(excitatory or inhibitory).</title></caption><alternatives><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"/><col align="center" span="1"/><col align="center" span="1"/></colgroup><thead><tr><td align="left" rowspan="1" colspan="1">Source/Dest.</td><td align="left" rowspan="1" colspan="1">Exc. (U, D, F)</td><td align="left" rowspan="1" colspan="1">Inh. (U, D, F)</td></tr></thead><tbody><tr><td align="left" rowspan="1" colspan="1">Exc.</td><td align="left" rowspan="1" colspan="1">0.5, 1.1, 0.02</td><td align="left" rowspan="1" colspan="1">0.25, 0.7, 0.02</td></tr><tr><td align="left" rowspan="1" colspan="1">Inh.</td><td align="left" rowspan="1" colspan="1">0.05, 0.125, 1.2</td><td align="left" rowspan="1" colspan="1">0.32, 0.144, 0.06</td></tr></tbody></table></alternatives></table-wrap> <table-wrap id="pcbi-1000180-t003" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.t003</object-id><label>Table 3</label><caption><title>Specific parameter values for the cortical microcircuits in computer
simulation 1 and 5.</title></caption><alternatives><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/></colgroup><thead><tr><td align="left" rowspan="1" colspan="1">Simulation No.</td><td align="left" rowspan="1" colspan="1">Neurons</td><td align="left" rowspan="1" colspan="1">
<italic>p<sub>ee</sub></italic>,
<italic>p<sub>ei</sub></italic>,
<italic>p<sub>ie</sub></italic>, <italic>p<sub>ii</sub></italic>
</td><td align="left" rowspan="1" colspan="1">
<italic>w<sub>exc</sub></italic>(0)
[nS]</td><td align="left" rowspan="1" colspan="1">
<italic>w<sub>inh</sub></italic>
[nS]</td><td align="left" rowspan="1" colspan="1">
<italic>C<sub>OU</sub></italic>
</td></tr></thead><tbody><tr><td align="left" rowspan="1" colspan="1">1</td><td align="left" rowspan="1" colspan="1">4000</td><td align="left" rowspan="1" colspan="1">0.02,0.02,0.024,0.016</td><td align="left" rowspan="1" colspan="1">10.7</td><td align="left" rowspan="1" colspan="1">211.6</td><td align="left" rowspan="1" colspan="1">1.0, 0.2</td></tr><tr><td align="left" rowspan="1" colspan="1">5</td><td align="left" rowspan="1" colspan="1">540</td><td align="left" rowspan="1" colspan="1">0.1</td><td align="left" rowspan="1" colspan="1">0.784</td><td align="left" rowspan="1" colspan="1">5.1</td><td align="left" rowspan="1" colspan="1">0.4</td></tr></tbody></table></alternatives></table-wrap> <table-wrap id="pcbi-1000180-t004" position="float"><object-id pub-id-type="doi">10.1371/journal.pcbi.1000180.t004</object-id><label>Table 4</label><caption><title>Specific parameter values for the trained (readout) neurons in
computer simulation 2, 4, and 5.</title></caption><alternatives><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"/><col align="center" span="1"/><col align="center" span="1"/><col align="center" span="1"/></colgroup><thead><tr><td align="left" rowspan="1" colspan="1">Simulation No.</td><td align="left" rowspan="1" colspan="1">Num. Synapses</td><td align="left" rowspan="1" colspan="1">
<italic>w<sub>max</sub></italic>
[nS]</td><td align="left" rowspan="1" colspan="1">
<italic>C<sub>OU</sub></italic>
</td></tr></thead><tbody><tr><td align="left" rowspan="1" colspan="1">2</td><td align="left" rowspan="1" colspan="1">100</td><td align="left" rowspan="1" colspan="1">11.9</td><td align="left" rowspan="1" colspan="1">1.0</td></tr><tr><td align="left" rowspan="1" colspan="1">4</td><td align="left" rowspan="1" colspan="1">200</td><td align="left" rowspan="1" colspan="1">5.73</td><td align="left" rowspan="1" colspan="1">0.2</td></tr><tr><td align="left" rowspan="1" colspan="1">5</td><td align="left" rowspan="1" colspan="1">432</td><td align="left" rowspan="1" colspan="1">2.02</td><td align="left" rowspan="1" colspan="1">0.2</td></tr></tbody></table></alternatives></table-wrap>
<disp-formula><label>(1)</label></disp-formula> <disp-formula><label>(2)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(3)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(4)</label></disp-formula> <inline-formula></inline-formula> <disp-formula><label>(5)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(6)</label></disp-formula> <disp-formula><label>(7)</label></disp-formula> <disp-formula><label>(8)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(9)</label></disp-formula> <inline-formula></inline-formula> <disp-formula><label>(10)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(11)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(12)</label></disp-formula> <inline-formula></inline-formula> <disp-formula><label>(13)</label></disp-formula> <disp-formula><label>(14)</label></disp-formula> <disp-formula><label>(15)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(16)</label></disp-formula> <disp-formula></disp-formula> <disp-formula><label>(17)</label></disp-formula> <inline-formula></inline-formula> <disp-formula><label>(18)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(19)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula><label>(20)</label></disp-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula><label>(21)</label></disp-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(22)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <inline-formula></inline-formula> <disp-formula><label>(23)</label></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula><label>(24)</label></disp-formula> <disp-formula><label>(25)</label></disp-formula> <disp-formula><label>(26)</label></disp-formula> <disp-formula><label>(27)</label></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula></disp-formula> <disp-formula><label>(28)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(29)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <disp-formula><label>(30)</label></disp-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula> <inline-formula></inline-formula>
<supplementary-material content-type="local-data" id="pcbi.1000180.s001"><label>Figure S1</label><caption><p>Variations of ##FIG##4##Figure
5B–D## for those excitatory neurons which receive the
full amount of Ornstein-Uhlenbeck noise. (B) The distribution of the firing
rates of these neurons remains unchanged during the simulation. The colors
of the curves and the corresponding intervals are as follows: red
(300–360 sec), green (600–660 sec), blue
(900–960 sec), magenta (1140–1200 sec). (C)
Cross-correlogram of the spiking activity of these neurons, averaged over
200 pairs of neurons and over 60 s, with a bin size of 0.2 ms, for the
period between 300 and 360 seconds of simulation time. It is calculated as
the cross-covariance divided by the square root of the product of variances.
(D) As in (C), but for the last 60 seconds of the simulation. The
correlation statistics in the circuit is stable during learning.</p><p>(0.06 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s002"><label>Figure S2</label><caption><p>Variations of ##FIG##4##Figure
5B–D## for those excitatory neurons which receive a
reduced amount of Ornstein-Uhlenbeck noise, but receive more synaptic inputs
from other neurons. (B) The distribution of the firing rates of these
neurons remains unchanged during the simulation. The colors of the curves
and the corresponding intervals are as follows: red (300–360 sec),
green (600–660 sec), blue (900–960 sec), magenta
(1140–1200 sec). (C) Cross-correlogram of the spiking activity in
the circuit, averaged over 200 pairs of these neurons and over 60 s, with a
bin size of 0.2 ms, for the period between 300 and 360 seconds of simulation
time. It is calculated as the cross-covariance divided by the square root of
the product of variances. (D) As in (C), but for the last 60 seconds of the
simulation. The correlation statistics in the circuit is stable during
learning.</p><p>(0.06 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s003"><label>Figure S3</label><caption><p>Variation of ##FIG##3##Figure 4##
from computer simulation 1 with results from a simulation where the
weight-dependent version of STDP proposed in ##REF##17444756##[22]## was used.
This STDP rule is defined by the following equations:  and . We used the parameters proposed in ##UREF##5##[36]##, i.e.
<italic>μ</italic> = 0.4,
<italic>α</italic> = 0.11,
<italic>τ</italic>
<sub>+</sub> = <italic>τ</italic>
<sub>−</sub> = 20
ms, <italic>λ</italic> = 0.1 and
<italic>w</italic>
<sub>0</sub> = 272.6
pS. The <italic>w</italic>
<sub>0</sub> parameter was calculated according to
the formula:  where <italic>w<sub>max</sub></italic> is the maximum
synaptic weight of the synapse. The amplitude parameters ,  for the reward kernel were set to  and . All other parameter values were the same as in computer
simulation 1.</p><p>(0.09 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s004"><label>Figure S4</label><caption><p>Variation of ##FIG##4##Figure 5## for
the weight-dependent STDP rule from ##REF##17444756##[22]## (as in ##SUPPL##2##Figure
S3##).</p><p>(0.06 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s005"><label>Figure S5</label><caption><p>Variation of ##FIG##6##Figure 7##
(i.e., of computer simulation 2) for a simulation where we used
current-based synapses without short-term plasticity. The post-synaptic
response had an exponentially decaying form , with
<italic>τ<sub>ε</sub></italic> = 5
ms. The value of the maximum synaptic weight was
<italic>w<sub>max</sub></italic> = 32.9 pA.
All other parameter values were the same as in computer simulation 2.</p><p>(0.17 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s006"><label>Figure S6</label><caption><p>Dependence of the learning performance on the noise level in computer
simulation 2. The angular error (defined as the angle between the weight
vector <bold>w</bold> of the trained neuron at the end of the simulation and
the weight vector <bold>w</bold>* of the neuron
<italic>μ</italic>*) is taken as measure for the learning
performance, and plotted for 9 simulations with different noise levels that
are given on the X axis (in term of multiples of the noise level chosen for
##FIG##6##Figure 7##). All other
parameters values were the same as in computer simulation 2. The figure
shows that the learning performance declines both for too little and for too
much noise.</p><p>(0.02 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s007"><label>Figure S7</label><caption><p>Variation of ##FIG##8##Figure 9##
(i.e., of computer simulation 4) with the weight-dependent STDP rule
proposed in ##REF##17444756##[22]##. This rule is defined by the following
equations:  and . We used the parameters proposed in ##REF##17444756##[22]##, i.e.
<italic>μ</italic> = 0.4,
<italic>α</italic> = 0.11,
<italic>τ</italic>
<sub>+</sub> = <italic>τ</italic>
<sub>−</sub> = 20
ms, <italic>λ</italic> = 0.1 and
<italic>w</italic>
<sub>0</sub> = 72.4
pS. The <italic>w</italic>
<sub>0</sub> parameter was calculated according to
the formula:  where <italic>w<sub>max</sub></italic> is the maximum
synaptic weight of the synapse. The amplitude parameters of the reward
kernel were set to
<italic>α<sub>P</sub></italic> = −<italic>α<sub>N</sub></italic> = 1.401.
All other parameter values were the same as in computer simulation 4. The
variance of the membrane potential increased for pattern <italic>P</italic>
from 2.35 (mV)<sup>2</sup> to 3.66 (mV)<sup>2</sup> (C), and decreased for
pattern <italic>N</italic> (D), from 2.27 (mV)<sup>2</sup> to 1.54
(mV)<sup>2</sup>.</p><p>(0.31 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s008"><label>Figure S8</label><caption><p>Variation of ##FIG##8##Figure 9## for
a simulation where we used current-based synapses without short-term
plasticity. The post-synaptic response had an exponentially decaying form , with
<italic>τ<sub>ε</sub></italic> = 5
ms. The value of the maximum synaptic weight was
<italic>w<sub>max</sub></italic> = 106.2 pA
All other parameter values were the same as in computer simulation 4. The
variance of the membrane potential increased for pattern <italic>P</italic>
from 2.84 (mV)<sup>2</sup> to 5.89 (mV)<sup>2</sup> (C), and decreased for
pattern <italic>N</italic> (D), from 2.57 (mV)<sup>2</sup> to 1.22
(mV)<sup>2</sup>.</p><p>(0.31 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s009"><label>Figure S9</label><caption><p>Variation of ##FIG##9##Figure 10##
(i.e., of computer simulation 5) for a simulation where we used
current-based synapses without short-term plasticity. The post-synaptic
response had an exponentially decaying form , with
<italic>τ<sub>ε</sub></italic> = 5
ms. The synaptic weights of the excitatory and inhibitory synapses in the
cortical microcircuit were set to
<italic>w<sub>exc</sub></italic> = 65.4 pA
and <italic>w<sub>inh</sub></italic> = 238
pA respectively. The maximum synaptic weight of the synapses to the readout
neuron was
<italic>w<sub>max</sub></italic> = 54.3 pA.
All other parameter values were the same as in computer simulation 5.</p><p>(0.27 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s010"><label>Figure S10</label><caption><p>Spike encodings of 10 utterances of digit “one” by one
speaker with the Lyon cochlea model ##UREF##6##[43]##, which were used
as circuit inputs for computer simulation 5.</p><p>(0.05 MB PDF)</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="pcbi.1000180.s011"><label>Figure S11</label><caption><p>Spike encodings of 10 utterances of digit “two” by one
speaker with the Lyon cochlea model ##UREF##6##[43]##, which were used
as circuit inputs for computer simulation 5.</p><p>(0.05 MB PDF)</p></caption></supplementary-material>
<table-wrap-foot><fn id="nt101"><p>These mean values, based on experimental data from ##REF##9560274##[37]##,##REF##10634775##[39]##, were
used in all computer simulations.</p></fn></table-wrap-foot> <table-wrap-foot><fn id="nt102"><p>
<italic>p<sub>conn</sub></italic> is the connection probability,
<italic>w<sub>exc</sub></italic>(0) and
<italic>w<sub>inh</sub></italic>(0) are the initial synaptic
weights for the excitatory and inhibitory synapses respectively, and
<italic>C<sub>OU</sub></italic> is the scaling factor for
the Ornstein-Uhlenbeck noise injected in the neurons.</p></fn></table-wrap-foot> <table-wrap-foot><fn id="nt103"><p>
<italic>w<sub>max</sub></italic> is the upper hard bound of the
synaptic weights of the synapses. <italic>C<sub>OU</sub></italic> is
the scaling factor for the Ornstein-Uhlenbeck noise injected in the
neurons.</p></fn></table-wrap-foot> <fn-group><fn fn-type="COI-statement"><p>The authors have declared that no competing interests exist.</p></fn><fn fn-type="financial-disclosure"><p>Written under partial support by the Austrian Science Fund FWF, project #
P17229-N04, project # S9102-N04, as well as project # FP6-015879 (FACETS) and
project # FP7-216886 (PASCAL2) of the European Union.</p></fn></fn-group>
<graphic xlink:href="pcbi.1000180.e001.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.g001"/> <graphic xlink:href="pcbi.1000180.e002.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e003.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e004.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e005.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e006.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e007.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e008.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e009.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e010.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e011.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e012.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e013.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e014.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e015.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e016.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e017.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e018.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e019.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e020.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e014.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e011.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e023.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e024.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e025.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e026.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e026.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g002"/> <graphic xlink:href="pcbi.1000180.g003"/> <graphic xlink:href="pcbi.1000180.e028.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e026.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g004"/> <inline-graphic xlink:href="pcbi.1000180.e030.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e031.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e032.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e031.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e032.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e031.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e036.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e037.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g005"/> <inline-graphic xlink:href="pcbi.1000180.e004.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e039.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e040.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e041.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e042.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e043.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g006"/> <graphic xlink:href="pcbi.1000180.e044.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e045.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e046.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e047.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e047.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e049.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e050.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e051.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e052.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e053.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e053.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e055.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e056.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e057.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e058.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e047.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e047.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e061.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e058.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e053.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e058.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e053.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e053.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e058.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e058.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e069.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g007"/> <inline-graphic xlink:href="pcbi.1000180.e070.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e071.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g008"/> <graphic id="pcbi-1000180-t001-1" xlink:href="pcbi.1000180.t001"/> <inline-graphic xlink:href="pcbi.1000180.e072.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e073.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e074.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e075.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e076.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e077.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e078.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e079.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e080.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e081.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e082.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e083.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e081.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.g009"/> <graphic xlink:href="pcbi.1000180.g010"/> <inline-graphic xlink:href="pcbi.1000180.e085.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e086.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e087.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e088.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e089.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e090.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e091.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e092.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e093.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e094.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e095.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e096.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e095.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e098.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e099.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e100.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e023.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e102.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e103.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e104.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e105.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e106.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e107.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e108.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e109.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e110.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e111.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e112.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e113.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e089.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e115.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e116.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e117.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e118.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e119.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e120.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e121.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e051.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e123.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e124.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e125.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e126.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e127.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e128.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e129.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e050.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e052.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e132.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e133.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e134.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e135.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e136.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e137.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e138.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e047.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e140.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e141.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e142.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e143.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e144.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e145.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e146.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e147.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e047.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e149.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e150.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e151.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e152.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e153.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e154.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e155.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e156.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e077.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e158.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e159.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e160.jpg" mimetype="image" position="float"/> <graphic id="pcbi-1000180-t002-2" xlink:href="pcbi.1000180.t002"/> <graphic xlink:href="pcbi.1000180.e161.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e162.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e163.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e164.jpg" mimetype="image" position="float"/> <graphic xlink:href="pcbi.1000180.e165.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e166.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e167.jpg" mimetype="image"/> <graphic id="pcbi-1000180-t003-3" xlink:href="pcbi.1000180.t003"/> <graphic id="pcbi-1000180-t004-4" xlink:href="pcbi.1000180.t004"/> <graphic xlink:href="pcbi.1000180.e168.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e169.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e170.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e171.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e172.jpg" mimetype="image"/> <graphic xlink:href="pcbi.1000180.e173.jpg" mimetype="image" position="float"/> <inline-graphic xlink:href="pcbi.1000180.e174.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e175.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e176.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e177.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e178.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e179.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e180.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e181.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e182.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e183.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e184.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e181.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e180.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e187.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e188.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e004.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e004.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e080.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e081.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e193.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e194.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e195.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e196.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e197.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e198.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e199.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e200.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e201.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e202.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e195.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e196.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e197.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e202.jpg" mimetype="image"/> <inline-graphic xlink:href="pcbi.1000180.e202.jpg" mimetype="image"/>
<media xlink:href="pcbi.1000180.s001.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s002.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s003.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s004.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s005.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s006.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s007.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s008.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s009.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s010.pdf"><caption><p>Click here for additional data file.</p></caption></media> <media xlink:href="pcbi.1000180.s011.pdf"><caption><p>Click here for additional data file.</p></caption></media>
[{"label": ["13"], "element-citation": ["\n"], "surname": ["Florian"], "given-names": ["RV"], "year": ["2007"], "article-title": ["Reinforcement learning through modulation of\nspike-timing-dependent synaptic plasticity."], "source": ["Neural Comput"], "volume": ["6"], "fpage": ["1468"], "lpage": ["1502"]}, {"label": ["14"], "element-citation": ["\n"], "surname": ["Baxter", "Bartlett"], "given-names": ["J", "PL"], "year": ["1999"], "article-title": ["Direct gradient-based reinforcement learning: I. Gradient\nestimation algorithms."], "comment": ["Technical report. Research School of Information Sciences and\nEngineering, Australian National University"]}, {"label": ["21"], "element-citation": ["\n"], "surname": ["Gerstner", "Kistler"], "given-names": ["W", "WM"], "year": ["2002"], "source": ["Spiking Neuron Models"], "publisher-loc": ["Cambridge, UK"], "publisher-name": ["Cambridge University Press"]}, {"label": ["34"], "element-citation": ["\n"], "surname": ["Maass", "Joshi", "Sontag"], "given-names": ["W", "P", "ED"], "year": ["2007"], "article-title": ["Computational aspects of feedback in neural circuits."], "source": ["PLoS Comput Biol"], "volume": ["3"], "fpage": ["e165"], "comment": ["doi:10.1371/journal.pcbi.0020165"]}, {"label": ["35"], "element-citation": ["\n"], "surname": ["Nikoli\u0107", "Haeusler", "Singer", "Maass"], "given-names": ["D", "S", "W", "W"], "year": ["2007"], "article-title": ["Temporal dynamics of information content carried by neurons in\nthe primary visual cortex."], "source": ["Proceedings of NIPS 2006, Advances in Neural Information Processing\nSystems. Volume 19"], "publisher-loc": ["Cambridge (Massachusetts)"], "publisher-name": ["MIT Press"], "fpage": ["1041"], "lpage": ["1048"]}, {"label": ["36"], "element-citation": ["\n"], "surname": ["Kempter", "Gerstner", "van Hemmen"], "given-names": ["R", "W", "JL"], "year": ["1999"], "article-title": ["Hebbian learning and spiking neurons."], "source": ["Phys Rev E"], "volume": ["59"], "fpage": ["4498"], "lpage": ["4514"]}, {"label": ["43"], "element-citation": ["\n"], "surname": ["Lyon"], "given-names": ["R"], "year": ["1982"], "article-title": ["A computational model of filtering, detection, and compression in\nthe cochlea."], "source": ["Proceedings of IEEE International Conference on ICASSP"], "fpage": ["1282"], "lpage": ["1285"]}, {"label": ["44"], "element-citation": ["\n"], "surname": ["Schrauwen", "Campenhout"], "given-names": ["B", "JV"], "year": ["2003"], "article-title": ["BSA, a fast and accurate spike train encoding scheme."], "source": ["Proceedings of the International Joint Conference on Neural Networks.\nVolume 4"], "fpage": ["2825"], "lpage": ["2830"]}, {"label": ["45"], "element-citation": ["\n"], "surname": ["Verstraeten", "Schrauwen", "Stroobandt", "Campenhout"], "given-names": ["D", "B", "D", "JV"], "year": ["2005"], "article-title": ["Isolated word recognition with the liquid state machine: a case\nstudy."], "source": ["Inf Process Lett"], "volume": ["95"], "fpage": ["521"], "lpage": ["528"]}]
PLoS Comput Biol. 2008 Oct 10; 4(10):e1000180