a critical problem in animal and human decision making is how to choose behavior that will in the long run lead to reward . for instance , when playing a game of chess , early moves may be crucial to the long - term goal of winning : one set of moves may set the stage for a stunning victory ; another result in a precipitous defeat .
the difficulty in learning to play chess stems from the fact that the outcome , i.e. , the win or loss at the end of the game , may be delayed with respect to such early actions .
moreover , once the game is over it is not clear , of the many actions taken throughout the game , which were the critical ones that should be learned for future games .
similar problems are encountered to a greater or lesser extent throughout our daily lives , in the many situations in which feedback for actions is not immediately available .
failure to solve this so - called  credit assignment  problem ( barto et al .
, 1983 ; sutton and barto , 1998 ) may be manifest as persistent , maladaptive decision - making : rather than being influenced by long - term goals , actions will be driven predominantly by proximal outcomes , thereby appearing impulsive and  irrational  , and succumbing easily to the temptations of short term pleasures .
one possible example of this is addiction , where behavior seems to be driven by the early and immediate positive experiences with drugs , despite the often remote or delayed adverse consequences of continued drug use .
although they do not play chess , rats encounter similar credit assignment problems in tasks that require multiple actions in order to obtain rewards or avoid punishments ( e.g. , when navigating a maze to a goal location or making a series of lever pressing and food - magazine approach actions in an operant chamber ) .
interestingly , exposure to addictive drugs has been shown to cause maladaptive decision - making in rats performing such tasks ( bechara and damasio , 2002 ; calu et al .
, 2007 ; jentsch et al . , 2002 ; nelson and killcross , 2006 ; schoenbaum and setlow , 2005 ; schoenbaum et al . , 2004 ; simon et al . , 2007 ; vanderschuren and everitt , 2004 )
. in a study recently published in frontiers in integrative neuroscience by takahashi et al .
( 2007 ) , these effects were linked to long - term changes in associative representations in two subdivisions of the striatum , the ventral striatum and the dorsolateral striatum .
these areas have previously been implicated in the neural instantiation of a computational solution to the credit assignment problem , called the actor / critic learning and decision - making architecture . in this focus article
, we discuss their results and offer a computational interpretation in terms of drug - induced modifications of the actor / critic model .
we begin by describing the actor / critic framework , and presenting a short survey of the different lines of evidence linking it to the ventral and dorsolateral striatum and their dopaminergic afferents .
( 2007 ) for such a model , by describing two possible scenarios in which the actor must behave in the absence of a functional critic .
one way to alleviate the credit assignment problem is by using as the advice of a coach : whether in sports or in chess , a coach who can give immediate feedback to actions even before the game has ended , can make all the difference in terms of learning .
what the player really needs is someone that can tell her whether her current action is good or bad .
importantly , for this information to be useful , it should evaluate the action with respect to the long - term goal of winning : a coach that reinforces a locally good action of capturing a pawn , despite the fact that a more strategic action had to be forfeited for this capture to happen , is rather dispensable .
the actor / critic architecture ( figure 1a ) does exactly this : the actor chooses actions according to some policy of behavior , and the critic offers immediate feedback that tells the actor whether the action selected was good or bad form the point of view of obtaining rewards in the long run ( barto et al . , 1983 ) .
the crux of this model is a simple learning rule based on a temporal difference ( td ) prediction error ( barto et al . , 1989 ; sutton , 1988 ; sutton and barto , 1990 ) : 
 t = r(st)+v(st)v(st1 ) , by which the critic learns what feedback to give to each action , and the actor learns an improved action selection policy . the components of this prediction error will be defined below . the basic actor / critic architecture and its suggested neural implementation .
( a ) the ( external or internal ) environment provides two signals to the system : s , indicating the current state or stimuli , and r indicating the current reward .
the actor comprises of a mapping between states s and action propensities (a|s ) ( through modifiable weights or associative strengths ) .
its ultimate output is an action which then feeds back into the environment and serves to ( possibly ) earn rewards and change the state of the environment .
the critic comprises of a mapping between states s and values v ( also through modifiable weights ) .
the value of the current state provides input to a temporal difference ( td ) module that integrates the value of the current state , the value of the previous state ( indicated by the feedback arrow ) and the current reward , to compute a prediction error signal t  =  r(st )  +  v(st )    v(st1 ) .
this signal is used to modify the mappings in both the actor and the critic .
( b ) a suggested mapping of the actor / critic architecture onto neural substrates in the cortex and basal ganglia .
the mapping between states and actions in the actor is realized through plastic synapses between the cortex and the dorsolateral striatum .
the mapping between states and their values is realized though similarly modifiable synaptic strengths in cortical projections to the ventral striatum .
the prediction error is computed in the ventral tegmental area ( vta ) and the substantia nigra pars compacta ( snc )  the two midbrain dopaminergic nuclei  based on state values from ventral striatal afferents , and outcome information from sources such as the pedunculopontine nucleus ( pptn ) , the habenula etc .
( christoph et al . , 1986 ; ji and shepard , 2007 ; kobayashi and okada , 2007 ; matsumoto and hikosaka , 2007 ) .
nigrostriatal and mesolimbic dopaminergic projections to the dorsolateral and ventral striatum , respectively , are used to modulate synaptic plasticity according to temporal difference learning . in order to compute such a feedback prediction error
, the critic must first evaluate the present state of the world ( st  the current situation , comprised of the available stimuli and context ) in terms of the long - term expected sum of future rewards when commencing behavior from this state .
that is , the critic can be seen as having two roles1 : one is to assign a value v(st ) to the current state , and the other is to combine state values with obtained rewards and compute the prediction error above .
as we shall show below , these two roles are intimately intertwined . in this reinforcement learning framework ,
state evaluations are essentially an estimation of the sum of future rewards from this point onward . as a result
, they should obey a simple local consistency relation : the reward expected from this time point onward should be equal to the immediate reward at the next time point plus any rewards expected from that time point onward , that is v(st1 )  =  r(st )  +  v(st ) where r(st ) denotes the ( possibly stochastic or 0 ) immediate reward at state st .
this consistency forms the basis for the td prediction error : t above quantifies the inconsistency between successive values , i.e. , the difference between reward predictions in consecutive states . to learn correct values ,
all the critic needs to do is to compute errors in its own value predictions and update its predictions such as to minimize these errors : 
 v(st1)new = v(st1)old+t 
 where  is a learning rate ( 0 
<    <  1 ; for a more detailed discussion of td learning , see barto , 1995 ; niv and schoenbaum , 2008 ) .
the elegance of this learning scheme is that local information ( immediate rewards and stored estimations of values of the current and immediately preceding states ) can be used to incrementally learn correct long - run value predictions .
armed with long - run predictive values , the critic can now coach the actor .
in fact , it turns out that the actor can also learn from the same prediction error that the critic computes in order to train its own values .
this is because all that the actor needs to know after every action , is whether this action improved its prospects for reward in the future , or not , that is , whether r(st )  +  v(st )  >  v(st1 ) , i.e. , the prediction error t is positive , or r(st )  +  v(st )  <  v(st1 ) and the prediction error t is negative , respectively .
actions that result in an increase in expected rewards should be repeated more often when in a similar situation , and vice versa for those which result in a decreased expectation of reward .
thus the same learning rule can be used to update the actor 's action propensities (a|st ) : 
 (a|st1)new=(a|st1)old+t where (a|st1 ) is the propensity to perform action a when in state st1 . to recap , in the actor / critic architecture the critic stores and learns state values and uses these to compute prediction errors , by which it updates its own state values .
these same prediction errors are also conveyed to the actor who stores and learns an action selection policy .
the actor uses the critic 's prediction error as a surrogate reinforcement signal with which it can improve its policy . in this particular division of labor ,
the  environment  sees only the output of the actor , ( i.e. , the actions ) , however , rewards from the environment are only of interest to the critic .
the actor / critic architecture is by no means the only solution to the credit assignment problem  it is certainly not the most efficient solution or computationally sound option ( see sutton and barto , 1998 for a variety of other reinforcement learning algorithms ) .
however , converging behavioral , anatomical and physiological evidence links the actor / critic architecture to habitual behavior of animals and humans , and to action selection mechanisms in the basal ganglia ( barto , 1995 ; houk et al . , 1995 ; joel et al . , 2002 )
the main neural structures implicated in associative learning and action selection are the basal ganglia ( in particular the striatum ) , limbic subcortical structures ( the amygdala and the hippocampus ) , and prefrontal cortical areas . generally speaking
, these can be viewed as the decision - making interface between sensory ( input ) and motor ( output ) areas of the brain .
the striatum , which is the input structure of the basal ganglia , receives convergent topographically organized inputs from motor , sensory , and limbic prefrontal cortical areas , as well as from the basolateral nuclei of the amygdala , the hippocampus , and the sensory thalamus .
the basal ganglia then provide a positive feedback loop to the frontal cortex through cortico - striatal - pallido - thalamo - cortical loops ( albin et al .
, 1989 ; alexander et al . , 1986 ; joel and weiner , 1994 ; parent and hazrati , 1993 ) .
thus the striatum is well positioned to identify meaningful associations between cues , responses and outcomes and to use this information to influence planning and action selection in the cortex ( joel and weiner , 1999 ) .
dopaminergic projections to the striatum are thought to play a critical role in this process by signaling reward prediction errors ( montague et al . , 1996 ; schultz et al . , 1997 ) and modulating plasticity in cortico - striatal synapses ( wickens , 1990 ; wickens et al . , 2003 ) .
early mappings of the actor / critic model onto the basal ganglia identified the value - learning part of the critic and the policy learning actor with the  patch  ( or striosome ) and  matrix  ( or matrisome ) subpopulations of striatal medium spiny neurons , respectively ( brown et al .
, 1999 ; contreras - vidal and schultz , 1999 ; houk et al . , 1995 ; suri , 2002 ; suri and schultz , 1999 ; suri et al . , 2001 ) , and the prediction error signal as carried by dopaminergic projections to the striatum ( e.g. , barto , 1995 ; houk et al . , 1995 ) .
the plausibility of the patch / matrix mapping was later challenged on anatomical grounds ( joel et al . , 2002 ) , specifically in relation to how values in the critic could affect the computation of a prediction error in dopaminergic nuclei , which is then conveyed to both critic and actor . instead , current findings suggest a mapping in which the ventral subdivision of the striatum ( including the nucleus accumbens ) embodies the value learning part of the critic that influences dopaminergic prediction errors , and the dorsal striatum is the actor ( joel et al . , 2002 ; but see atallah et al . , 2007 for a slightly different suggestion ) . while much evidence supports this suggestion ,
still more recent data advocate a refinement of the actor to include only the dorsolateral striatum ( with the dorsomedial striatum implicated in a different form of goal - directed action selection that can not be supported computationally by an actor / critic architecture ; balleine , 2005 ; yin et al . , 2005 ) . in the following
we will briefly review the evidence supporting this ventral  dorsolateral striatal mapping of the actor / critic model , depicted with more detail in figure 1b .
recent work on the striatum has emphasized its division into different substructures based on connectivity patterns and functional roles ( see figure 2 ) .
the primary division is into dorsal and ventral parts ( or dorsolateral and ventromedial ; voorn et al . , 2004 ) .
the dorsal striatum is further comprised of two subparts : the caudate nucleus ( or its homologue in rats , the dorsomedial striatum ) and the putamen nucleus ( or the dorsolateral striatum in rats ) . similarly , within the ventral striatum the nucleus accumbens can be subdivided into core and shell compartments .
this anatomical parcellation is in line with a previously suggested functional division of the basal ganglia into limbic ( accumbal ) , associative ( dorsomedial striatal ) and sensorimotor ( dorsolateral striatal ) loops ( joel and weiner , 1994 ; parent and hazrati , 1993 ) .
mesolimbic dopamine  ) target the ventral striatum ( including the nucleus accumbens ) while  nigrostriatal 
dopamine arising from the substantia nigra pars compacta ( snc ) targets the dorsal striatum ( amalric and koob , 1993 ) .
anatomical organization of the rat striatum and its afferents . in the center is a transverse section of the rat forebrain showing the striatum .
converging excitatory afferents from the frontal cortex ( upper left ) , midline and intralaminar thalamic nuclei ( upper right ) , basal amygdaloid complex ( lower left ) and hippocampal formation ( lower right ) project topographically to dorsomedial - to - ventrolateral zones .
frontal cortical areas and their corresponding striatal projection zones are shown in the same colors .
abbreviations : acc , anterior cingulate cortex ; il , infralimbic cortex ; pfc , prefrontal cortex ; dpl and vpl , dorsal and ventral prelimbic cortices ; cem , cl , imd , md , pc and pv , central medial , central lateral , intermediodorsal , mediodorsal , paracentral and paraventricular thalamic nuclei , respectively ( adapted from voorn et al . , 2004 ) .
the ventral striatum , at the heart of the limbic corticostriatal loop , is well positioned to support learning of predictive values ( as in the critic ) .
afferents from so - called  limbic  areas , the basolateral amygdala and hippocampus , as well as from prefrontal cortical areas ( voorn et al . ,
2004 ) , can convey information about the current context and stimuli , and their affective values .
efferents from the ventral striatum , in turn , target dopaminergic neurons , thus state values can be incorporated into the prediction error signal .
importantly , the ventral striatum influences activity in both dopaminergic nuclei , that is , it projects to those dopaminergic neurons that project back to the ventral striatum and to those that project to the more motor - related dorsal striatum ( haber et al . , 1990 , 2000 ; joel and weiner , 1999 , 2000 ; lynd - balta and haber , 1994 ; nauta et al . , 1978 ) . due to this strategic positioning
in terms of the suggested actor / critic mapping , the ventral striatum is in a prime position to influence prediction error learning signals for both the actor and the critic , as is necessary in the model ( see figure 1 ) . in accord with these suggestive anatomical data ,
behavioral , pharmacological , and neural imaging results have implicated the ventral striatum in evaluation and prediction , and the dorsolateral striatum in action selection and execution ( e.g. , o'doherty et al . , 2004 ; pessiglione et al .
more specifically , the ventral striatum has been shown to be involved in reward anticipation , in attributing value to pavlovian stimuli , and in mediating the ability of the affective value of anticipated outcomes to affect instrumental performance , as in paradigms such as pavlovian - instrumental transfer and conditioned reinforcement ( balleine , 2005 ; cardinal et al . , 2002 ; everitt et al .
a further division of labor between the nucleus accumbens core and shell , putatively involved in general value predictions ( as in the critic ) and specific outcome values ( as is necessary for correct goal - directed responding ) , respectively , is also in line with current data ( cardinal et al . , 2002 ; corbit et al .
the dorsolateral striatum , on the other hand , has been implicated in habit learning and in the control of habitual responding ( atallah et al .
, 2007 ; bailey and mair , 2006 ; balleine , 2005 ; knowlton et al .
, 1996 ; racht - delatour and el massioui , 2000 ; yin et al . , 2004 , 2006 ) , and is also the major site of dopamine loss in idiopathic parkinson 's disease , with its predominantly motor - related symptoms ( e.g. , kish et al . , 1988 ) . finally , recent results show that disconnecting the dopaminergic influence of the accumbens core on the dorsolateral striatum impairs the execution of habitual responding for a second order reinforcer predictive of cocaine ( belin and everitt , 2008 ) . this effect may be general and not limited to drug - seeking behavior , thus illustrating the importance of these spiraling projections from ventral to dorsal areas .
it has proven more difficult to garner direct support for this hypothesized functional subdivision from electrophysiological studies , as these have uncovered a bewildering wealth of representations in the striatum  essentially every possible task - related event can be shown to be represented by striatal medium spiny neurons ( e.g. , schultz et al .
still , prominent in these are representations of outcome anticipation in ventral striatal areas ( schultz et al . , 1992 ; tremblay et al .
, 1998 ; williams et al . , 1993 ) and those of actions and of action  chunks  in the dorsal striatum ( kimura , 1986 , 1995 ; jog et al . , 1999 ;
structurally , glutamatergic cortical afferents and midbrain dopaminergic inputs converge onto common dendrites of striatal medium spiny neurons .
functionally , and important for the actor / critic model , dopamine has been shown to affect plasticity in cortico - striatal synapses , predominantly through d1 receptors , forming what has been termed a  three factor learning rule  in which the coincidence of presynaptic and postsynaptic activity causes ltp if a modulatory dopaminergic signal is also present , and ltd otherwise ( calabresi et al . , 2000 ; houk et al .
, 1995 ; wickens , 1990 ; wickens and kotter , 1995 ; wickens et al . , 1996 ) .
for instance , in the accumbens core , co - infusion of low doses of a d1 receptor antagonist and an nmda blocker impaired acquisition of instrumental learning ( lever pressing for food ) , even though these low doses were not sufficient to induce an impairment when infused alone ( smith - roe and kelley , 2000 ) . in sum , the evidence is consistent with a prediction and action selection system in which phasic dopamine signals a td prediction error signal , which trains value predictions ( as in the critic ) via projections from the vta to ventral striatal and frontal target areas while simultaneously reinforcing the current action - selection policy ( as in the actor ) through snc projections to the dorsolateral striatum .
indeed , within a broader scheme , one might view the basal - ganglia / dopamine system as a system of multiple actors and critics operating in parallel , which can collaborate or compete to control behavior ( daw et al . , 2005 ) . in this
, the dorsolateral striatum may support an actor that relies on action propensities that are learned incrementally through trial and error ( as described above ) , suggestive of inflexible stimulus - response habits that are independent of specific outcome predictions .
by contrast , the dorsomedial striatum may support a second actor that relies on computations of action values based on so - called response - outcome representations , thereby allowing goal - directed action selection that is more flexible and controlled by detailed outcome expectations ( and not only scalar valued expectations in some common currency ) .
finally , pavlovian responding may result from the operation of a third actor or may be a direct consequence of value predictions ( dayan et al . , 2006 ) , thus placing the pavlovian  actor  in value learning areas such as the ventral striatum and the amygdala ( balleine and killcross , 2006 ) .
whether the value learning critic system is also divisible into a scalar prediction critic ( as necessary for the traditional actor / critic architecture ) , perhaps in the nucleus accumbens core and central nucleus of the amygdala , and a second critic supporting more specific predictions of outcomes ( as in goal - directed behavior ) , perhaps in the accumbens shell , basolateral amygdala and/or orbitofrontal cortex , is less clear2 . also unclear is the role of dopamine in training the non - habitual actors , and indeed whether there are outcome - specific and outcome - general prediction errors . in any case , for our current purposes the rather well - substantiated mapping of the dorsolateral striatum to a habitual actor , and the nucleus accumbens core to the scalar value learning critic ( that can also support preparatory pavlovian responses ) suffices .
( 2007 ) , to which we will now turn , were from these two areas .
if indeed the dorsolateral and ventral striatum and their corresponding dopaminergic signals function in such a highly integrated manner as suggested by the actor / critic architecture ( for challenges to this view , see dayan and balleine , 2002 ) , then small perturbations could be expected to have pronounced effects on learning and ultimately on normal , adaptive ( habitual ) responding . in the next section , we briefly describe evidence from takahashi et al .
( 2007 ) showing that exposure to an addictive drug causes massive changes in putative correlates of associative learning in this system .
thereafter we will consider these changes within an actor / critic architecture and use this model to suggest potential mechanisms whereby cocaine might be having its effects .
to study the long - term effects of cocaine on signaling of predicted value in the striatum , takahashi et al .
( 2007 ) recorded single - unit activity in the ventral and dorsolateral striatum of rats that had been sensitized to cocaine approximately 13 months earlier ( figure 3a ) .
recordings were made as these rats and saline - treated control animals learned and reversed a series of novel go , no - go odor discrimination problems ( figure 4a ) . in each problem , a pair of novel odor cues were paired with a sucrose reward and a quinine punishment ( schoenbaum et al . , 1999b ) .
correct performance depended on acquiring associations between the odor cues and the rewarding and aversive outcomes , presumably resulting in the attribution of motivational value or significance to the cues .
in addition , the rats were required to associate each cue with a different response ( go or no - go ) in order to solve the discrimination and each reversal ( figure 4a ) .
previous work has shown that rats sensitized to or trained to self - administer cocaine exhibit a long - lasting reversal learning impairment in this task ( calu et al .
consistent with this , cocaine - treated rats in the current study successfully reversed a much lower proportion of the odor problems to which they were exposed than the saline - treated controls .
single - unit activity in ventral striatum in saline- and cocaine - treated rats during discrimination and reversal learning .
( a ) experimental timeline showing the relative timing of behavioral training , cocaine sensitization and recording .
( b , c ) average activity in populations of neurons recorded in ventral striatum ( nucleus accumbens core ) during acquisition and reversal of a series of odor discrimination problems ( only successful reversals were included ) , during pre - criterion trials , post - criterion trials and post - reversal .
the left column ( b ) shows activity in saline - treated control rats and the right column ( c ) shows activity in cocaine - treated rats . in each case ,
population responses are shown separately for all neurons ( small insets ) and for cue - selective neurons ( large plots ; cue - selective neurons were defined as any neuron with differential activity during odor sampling after learning at p  <  0.05 , anova ) . in blue is activity on trials with odor 1 ( which predicted sucrose before reversal ) , and in red is activity on trials with odor 2 ( which predicted quinine before reversal ) .
bar graphs show averaged firing rates ( spikes per second ) for each odor in each phase in the cue - selective populations ( * , significant difference at p  <  0.05 ; * * , significant difference at p  <  0.01 or better , anova ) .
the neural population in saline - treated controls showed strong firing to the odor cue that predicted the aversive quinine outcome in pre - criterion trials , post - criterion trials , and after reversal .
in contrast , the neural population in cocaine - treated rats was not cue - selective in either phase ( adapted from takahashi et al . , 2007 ) .
single - unit activity in dorsolateral striatum in saline- and cocaine - treated rats during discrimination and reversal learning .
( a ) timing of events within each trial . after reversal , the mapping between odor 1 and odor 2 and the sucrose and quinine outcomes was reversed .
( b , c ) average activity in populations of neurons recorded in dorsolateral striatum during acquisition and reversal of a series of odor discrimination problems ( conventions as in figure 3 ) .
the neural population in cocaine - treated group exhibited stronger phasic firing to the odor cue that predicted the sucrose outcome than the population in saline - treated controls .
this difference was especially evident during pre - criterion and post - criterion trials ( adapted from takahashi et al . , 2007 ) .
( 2007 ) found that in control rats neurons in both ventral and dorsolateral striatum fired to the cues based on their associations with subsequent responses and/or outcomes .
this cue - selective activity developed with learning and reversed in both regions when the meaning of the cues was switched ( figures 3b and 4b ) .
however several key differences emerged between the two areas , consistent with their putative roles in the actor / critic models .
cue - selectivity in ventral striatum developed rapidly during learning , preceding the development of accurate responding , and firing was much higher to the odor cue that predicted the more motivationally - significant aversive quinine outcome ( figure 3b ) .
in contrast , cue - selectivity in dorsolateral striatum developed later , only alongside accurate differential responding ( figure 4b ) .
these differences fit well with the proposal that ventral striatum learns the value of the cue , so that it is then able to coach the dorsolateral striatum regarding the proper action to execute in response to each odor cue .
consistent with the proposal that addiction might involve changes in striatal function , prior cocaine - treatment had a significant impact on the neural correlates in both striatal regions ( figures 3c and 4c , compare to figures 3b and 4b ) .
first , after learning , cocaine - treated rats had substantially fewer cue - selective neurons in ventral striatum than saline controls .
the absence of such neurons in cocaine - treated rats was evident in the population responses ( figure 3c ) , which did not exhibit cue - selectivity in any phase of training .
by contrast , cocaine - treatment caused a small but significant increase in the number of cue - responsive neurons in dorsolateral striatum , especially to the positive odor cue that drove most of the neural activity observed in controls .
the effect is somewhat evident in the population responses in figure 4c , where the relatively weak and slow to develop response to the positive odor cue seen in controls is more robust in the cocaine - treated rats .
the stronger response to the positive odor in cocaine - treated rats was particularly evident in the pre - criterion trials , thus differential activity appeared earlier in cocaine - treated rats in dorsolateral striatum .
these results show that cocaine sensitization shifted the balance of encoding between these two regions , abolishing the strong cue - selectivity normally present in ventral striatum while marginally enhancing the relatively weak cue - selectivity normally present in dorsolateral striatum . in the next section
, we will consider how such an effect might be interpreted within the actor / critic model of striatal function .
according to the above mapping between an actor / critic architecture and the subparts of the striatum , one consequence of radically reduced cue selectivity in the ventral striatum may be the severe degradation ( or even elimination ) of the feedback from the critic to the actor .
in other words , if cue - evoked activity in ventral striatum is in fact signaling information about state values , as proposed for the critic , then takahashi et al . 's ( 2007 ) results show that cocaine sensitization disrupts this function . within an actor / critic model
, this would have profound effects on the ability of the actor to learn to select the appropriate actions . in the following ,
we discuss two different ways that such a disturbance might be manifest , and their implications for action selection . in the discussion
we further develop ideas for future experiments targeted at testing these two alternatives more directly .
the simplest possibility , depicted in figure 5a , is that the critic fails to learn or to represent state values properly , that is , the v(st ) signals that it produces are meaningless .
these would then be integrated into the prediction error that trains the actor , and would lead to a perturbed learning process .
( 2007 ) in that cue - evoked activity in the ventral striatum of cocaine - sensitize rats did not distinguish between the two cues .
we can thus model this as a situation in which all v(st ) are the same , regardless of the state st . in this case
, the prediction error t  =  r(st )  +  v(st )    v(st1 ) reduces to t  =  r(st ) , i.e. , the prediction error is equal to the immediately available reward only .
( a ) the critic is unable to learn or represent a meaningful mapping between states and their values ( depicted is the extreme case of similar values for all states ) , thus the prediction error signal  that is used to train the actor comprises only of the current reward .
( b ) a deficient prediction error signal disrupts learning in both the critic and the actor ( depicted is the extreme case of no prediction error whatsoever ) .
although it is obviously nave to envision cocaine sensitization as completely wiping out any representation of v(st ) , we think it is useful to consider such an extreme situation in order to understand the potential consequences of a deficit in the critic 's representations .
specifically , we 'd like to ask what are the consequences of this hypothesized deficient training signal for the actor , i.e. , for representations in the dorsolateral striatum and for overt behavior ?
on the one hand , in tasks with outcomes that depend on a series of actions , that is , those scenarios in which the credit assignment problem is severe and an intact actor / critic architecture is necessary for proper learning , a learning signal that is based at every time - point only on immediate reward will not be sufficient to learn the task .
this predicts that cocaine sensitization would have a large detrimental effect on subsequent habitual learning and action selection in such tasks .
on the other hand , in those ( simpler to learn ) tasks in which actions are immediately rewarded in an unambiguous way , learning may actually be faster .
that is because it will not be hindered by incorrect estimations of future values which are common early on in the learning process .
odors predict the immediate outcome ( sucrose or quinine ) to a go or a no - go response . in this sense
, it is perhaps not surprising that performance was not disrupted after cocaine sensitization , and that representations in the dorsolateral striatum were intact , or even slightly enhanced . indeed in a prior study in which initial learning was carefully examined , cocaine - treated rats showed a small but nearly significant improvement in acquisition of the initial discrimination problem learned ( schoenbaum et al . , 2004 ) , and we have observed a similar facilitation in accumbens - lesioned rats ( schoenbaum and setlow , 2003 ) .
while such facilitations are difficult to observe in recording work , where rats have developed a  learning set  for these problems , these prior observations are consistent with the idea that removal of the part of the critic that is located in ventral striatum may actually enhance learning of simple discriminations .
furthermore , that cocaine - sensitized rats reversed fewer of the discriminations successfully may , in fact , hint at stronger learning of the initial discrimination , which then impeded later reversal of the behavioral policy . indeed , learning with a persistent reward signal ( i.e. , one that does not come to be
 predicted away  by predictive cues ) could lead to stronger habitual learning , to the point of maximal action propensities .
this idea would be consistent with other evidence showing that reversal deficits after cocaine are associated with inflexible encoding in the basolateral amygdala ( stalnaker et al . , 2007 ) .
a second option , depicted in figure 5b , is that cocaine sensitization affects the prediction error signal itself .
an invalid error signal ( or , considering once again the extreme case  no prediction error signal ) would disrupt learning in both the critic and the actor and thus would account for the loss of cue - selectivity in the ventral striatum
. however , this idea is not in line with the finding that both behavior and the representations in the dorsolateral striatum were intact , unless there is some other means by which action propensities and dorsolateral striatal representation can be learned , at least in cases in which rewards are immediate .
this latter suggestion is not entirely speculative  as mentioned , recent work has highlighted the existence of multiple learning and action selection systems in the brain , presumably all working in parallel and interacting ( whether in competition or in synergy ) to control behavior ( balleine 2005 ; daw et al . , 2005 ; johnson et al . , 2007 ) .
pavlovian  approach  and  withdrawal  responses , perhaps acquired through prediction learning in the central and basolateral nuclei of the amygdala , and/or in the nucleus accumbens shell ( from which takahashi et al . ,
2007 did not record ) , could suffice to generate the correct behavioral response ( dayan et al . , 2006 ) , and could perhaps provide surrogate training signals for the actor .
such alternative training could conceivably result in representations that are more resistant to reversal , due to the absence of a flexibly adapting dopaminergic training signal .
of the two options we suggest , it is not immediately clear which is more in line with the effects of cocaine sensitization .
this is partly due to the requirements of the task used by takahashi et al .
( 2007 ) , a go / no - go odor discrimination task , with immediate outcomes in each trial .
this task was chosen because rats can readily learn it ( and perform reversals ) , which allows for clear assessment of behavioral and neural implications of different treatments on the learning process itself .
indeed , this task has been widely used to record in other areas , and robust behavioral and neural effects have been shown with previous pharmacological or lesion manipulations ( gallagher et al . , 1999 ; saddoris et al . , 2005 ;
setlow et al . , 2003 ; stalnaker et al . , 2006 , 2007 )
however , that this task is learned with such ease is perhaps because of its simple structure : only one action is required to obtain ( or avoid ) an outcome ( see figure 4a ) , rather than a series of actions . moreover ,
this action is not opposed to the pavlovian tendency to approach predictors of reward and avoid predictors of punishment .
this is likely also the reason that cocaine - sensitized rats with severely disrupted ventral striatal representations were still able to learn new discriminations and reversals , and basically display intact behavior in this task .
indeed , it seems that in tasks that provide a more difficult test - bed for the actor / critic mechanism , both deficits should show more severe implications for learning and action selection , much beyond those seen in takahashi et al . 's ( 2007 ) results .
. 's ( 2007 ) study showed that cocaine sensitization disrupts encoding in the ventral striatum , while leaving dorsolateral striatal encoding and behavior mostly intact .
based on the idea that the dorsolateral and ventral striatum may function as an actor and a critic , respectively , we have suggested two ways in which these results can be interpreted within an actor / critic framework . although this is certainly not the only framework within which to consider these data , the actor / critic framework has strong empirical support and also makes very specific predictions about the underlying mechanism . in particular , the two ways that we suggest these changes might arise differ in whether the primary ( and thus treatable ) cause lies within ventral striatum itself or is in the incoming dopaminergic projections .
first , if ventral striatal representations are indeed used to generate a training signal for the dorsolateral striatum which takes into account delayed as well as immediate rewards , a task in which outcomes are dependent on a sequence of actions should show grossly disrupted learning and performance after cocaine sensitization . at the very least , even if behavioral impairments are masked by other ( intact ) action selection systems in the brain , we expect to see disrupted representations in the dorsolateral striatum in this case .
second , recordings from dopaminergic neurons should reveal an altered prediction error signal after cocaine sensitization : the signal should either be dominated by immediate rewards ( according to the option in figure 5a ) or be altogether uninformative or absent ( as in figure 5b ) .
third , if the prediction error signal is indeed wholly disrupted as in the second option above , a task in which pavlovian conditioning is not sufficient to produce the correct behavior could reveal additional behavioral deficits . finally , in this case ( as well as in general )
, it would be extremely interesting to test the implications of cocaine sensitization on representations in the shell of the nucleus accumbens and the dorsomedial striatum .
as we have already noted , cocaine - induced impairments in reversal tasks or in tasks that model aspects of reversal learning appear to reflect not an absence of learning but rather abnormally strong initial learning .
that is , addicts and animals exposed to cocaine show normal responding before reversal but then fail to change behavior after reversal . in the present study
, these reversal deficits seemed to be linked to enhanced signaling in the dorsolateral striatum .
however , we have previously demonstrated that they are also associated with persistent miscoding of the original associations in the basolateral amygdala ( stalnaker et al . , 2007 ) .
moreover , removal of the basolateral amygdala is sufficient to restore normal performance ( stalnaker et al . , 2007 ) , suggesting that this aberrant encoding is the proximal cause of the cocaine - dependent reversal deficit .
these results are also consistent with the observation that cue - evoked relapse is ameliorated by infusions of agents into the amygdala that disrupt memory reconsolidation during exposure to drug cues ( lee et al . , 2005 ) .
thus , drug exposure does not seem to prevent learning in general , as would be expected if error signals were eliminated .
instead , it seems to make learning more difficult when delayed or probabilistic outcomes must be tallied .
for instance , addicts and drug - treated animals show increased sensitivity to delay of reward , behaving more 
impulsively  , and also show increased sensitivity to reward magnitude ( coffey et al .
, 2003 ; roesch et al . , 2007 ; simon et al . , 2007 ) .
this pattern of results suggests interference with the critic , but supports the idea that at least some part of the prediction error signal is intact , and continues to support learning .
in fact , reliance on a prediction error that is dominated by immediate rewards would indeed result in increased impulsivity and sensitivity to reward magnitude .
however , these are also consistent with increasing reliance on a pavlovian learning system , which could be an obvious consequence of any type of disruption of the instrumental action - selection mechanism . interestingly , although it is not well - modeled with current theories of instrumental choice , cocaine also appears to cause significant changes in how information about expected outcomes is signaled by frontal areas .
we have previously demonstrated that cocaine - treated rats are unable to use information about outcomes to guide behavior in a pavlovian setting ( schoenbaum and setlow , 2005 ) , and killcross and colleague have shown analogous effects of amphetamine on outcome - guided behavior in an instrumental setting ( nelson and killcross , 2006 ) .
similar deficits in the control of pavlovian and instrumental behavior are caused by damage to the orbitofrontal and medial prefrontal areas respectively ( corbit and balleine , 2003 ; gallagher et al .
, 1999 ; killcross and coutureau , 2003 ) , and are thought to reflect the role of these areas in signaling information about expected outcomes , potentially to both instrumental and pavlovian systems .
thus drug exposure may also disrupt outcome signaling by these critical prefrontal areas . in accord with this hypothesis ,
we have found that orbitofrontal neurons in cocaine - treated rats fail to signal expected outcomes during cue - sampling ( stalnaker et al . , 2006 ) .
this proliferation of drug - induced deficits suggests yet a third possible explanation to the results we have described above : the loss of differential encoding of cues in the ventral striatum could be the result of disrupted representations carried by its cortical afferents .
thus , a common cause might be at the heart of the orbitofrontal , amygdalar , and ventral striatal deficits , or one of these may be influencing the others , to their detriment .
however , the preservation or even enhancement of the dorsolateral striatal representations argues against this idea , especially if the core of the deficit were in higher cortices that presumably modulate not only the limbic corticostriatal loop , but also the associative and motor loops ( joel and weiner , 1999 ) .
that the dorsolateral striatal representations remained intact thus suggests that the problem may not be as severe as an insidious lack of representation in the prefrontal cortex .
still , the relationships and interactions between the different drug - induced deficits remain as an intriguing question for future research . in sum ,
the pervasive and detrimental effects of ( even mild forms of ) drug abuse are astounding .
we have suggested here that combining experimental findings with a computational analysis can allow not only for a system - level interpretation of electrophysiological results and their relationship to behavioral deficits , but can also suggest a wealth of new experiments .
indeed , it sometimes seems that the questions outnumber the answers . by bringing theoretical models to bear on concrete findings , we can only hope to make another small step in the direction of understanding ( and ultimately curing ) drug addiction .
the authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest .
 a critical problem in daily decision making is how to choose actions now in order to bring about rewards later . indeed , many of our actions have long - term consequences , and it is important to not be myopic in balancing the pros and cons of different options , but rather to take into account both immediate and delayed consequences of actions . 
 failures to do so may be manifest as persistent , maladaptive decision - making , one example of which is addiction where behavior seems to be driven by the immediate positive experiences with drugs , despite the delayed adverse consequences . a recent study by takahashi et al . ( 2007 ) investigated the effects of cocaine sensitization on decision making in rats and showed that drug use resulted in altered representations in the ventral striatum and the dorsolateral striatum , areas that have been implicated in the neural instantiation of a computational solution to optimal long - term actions selection called the actor / critic framework . in this focus article 
 we discuss their results and offer a computational interpretation in terms of drug - induced impairments in the critic . 
 we first survey the different lines of evidence linking the subparts of the striatum to the actor / critic framework , and then suggest two possible scenarios of breakdown that are suggested by takahashi et al . 's ( 2007 ) data . 
 as both are compatible with the current data , we discuss their different predictions and how these could be empirically tested in order to further elucidate ( and hopefully inch towards curing ) the neural basis of drug addiction . 
Introduction
The Actor/Critic Model: a Player and a Coach
Actor/Critic in the Brain: the Basal Ganglia
Cocaine Sensitization: Differential Effects on the Dorsal/Ventral Divide
A Coach Gone Awry and an Actor Running Loose
Discussion
Conflict of Interest Statement