Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release
Lesaint and colleagues 1 recently proposed a new computational model—the “STGT model” ( for sign tracking and goal tracking ) —which accounts for a large set of behavioral , physiological , and pharmacological data obtained from studies investigating individual variation in Pavlovian conditioned approach behavior 2–8 ., Most notably , the model can account for recent work by Flagel and colleagues ( 2011 ) that has shown that phasic dopamine ( DA ) release does not always correspond to a reward prediction error ( RPE ) signal arising from a classical model-free ( MF ) system 9 ., In their experiments , Flagel and colleagues trained rats on a classical autoshaping procedure , in which the presentation of a retractable-lever conditioned stimulus ( CS; 8 s ) was followed immediately by delivery of a food pellet ( unconditioned stimulus US ) into an adjacent food cup ., In procedures like this , some rats , known as sign trackers ( STs ) , learn to rapidly approach and engage the CS lever , whereas other rats , known as goal trackers ( GTs ) , learn to approach and enter the food cup upon presentation of the CS lever ., Although both sign and goal trackers learn the CS-US relationship equally well , it was elegantly shown that phasic DA release in the nucleus accumbens core ( NAc ) matched RPE signals only in STs 4 ., Specifically , during learning in ST rats , DA release to reward decreased , while DA release to the CS increased ., In contrast , even though GTs acquired a Pavlovian conditioned approach response , DA release to reward did not decline , and CS-evoked DA was weaker ., Furthermore , administration of a DA antagonist blocked acquisition of the ST conditioned response but did not impact the GT conditioned response 4 , 10 ., Several computational propositions have argued that these data could be interpreted in terms of different contributions of model-based ( MB ) —with an explicit internal model of the consequences of actions in the task—and MF—without any internal model—reinforcement learning ( RL ) in GTs and STs during conditioning 1 , 11 ., Nevertheless , only the STGT model predicted that manipulating the intertrial interval ( ITI ) should change DA signaling in these animals: the model suggests that GTs revise the food cup value multiple times during and in between trials during the 90-s ITI ., During the trial , the food cup gains value because reward is delivered; however , visits to the food cup during the ITI do not produce reward , thus reducing the value assigned to the food cup ., This mechanism prevents the progressive transfer of reward value signal in the model from US time to CS time and hence explains the absence of DA RPE pattern in goal trackers ., This aspect of the model predicts that decreasing the ITI should reduce the amplitude of US DA burst ( i . e . , less time to negatively revise the value of the food cup and reduce the size of the RPE ) and that higher food cup value should lead to an increase in the tendency to GT in the overall population ., In contrast , increasing the ITI should have the opposite effect ., That is , lengthening the ITI and therefore increasing the number of nonrewarded food cup entries should increase the amplitude of US DA burst ( i . e . , more time to negatively revise the value of the food cup during the ITI and increase the size of the RPE ) and lower the value of the food cup , leading to a decreased tendency to GT and an increase tendency to ST . The latter would be accompanied by a large phasic DA response to the highly salient lever CS , as previously observed in STs 4 ., Here , we tested these predictions by recording DA release in the NAc using fast-scan cyclic voltammetry ( FSCV ) during 10 d of Pavlovian conditioning in rats that had either a short ITI of 60 s or a long ITI of 120 s ., DA release was recorded from NAc ( S1B–S1E Fig ) during a standard Pavlovian conditioned approach behavior task ( S1A Fig ) for 10 d ., Each trial began with the presentation of a lever ( CS ) located to the left or right side of a food cup ( counterbalanced ) for 8 s ., Upon the lever’s retraction , a 45-mg sucrose pellet was delivered into the food cup , independent of any interaction with the lever ., Each behavioral session consisted of 25 trials presented at a random time interval of either 60 s ( n = 7 rats ) or 120 s ( n = 12 rats ) ., To quantify the degree to which rats engaged in sign- versus goal-tracking behavior , we used the Pavlovian Conditioned Approach ( PCA ) index 12 , which comprised the average of three ratios: ( 1 ) the response bias , which is ( Lever Presses − Food Cup Entries ) / ( Lever Presses + Food Cup Entries ) , ( 2 ) the probability ( P ) difference , which is ( Plever − Preceptacle ) , and ( 3 ) the latency index , which is ( x¯ Cup Entry Latency − x¯ Lever Press Latency ) / 8 ., All of these ratios range from −1 . 0 to +1 . 0 ( similarly for PCA index ) and are more positive and negative for animals that sign track and goal track , respectively ., All behavioral indices were derived from sessions during which DA was recorded ., For the initial analysis described in this section , behavior and DA were examined across all sessions; the development of behavior and DA over training is examined in later sections ., The distributions of behavioral session scores are shown in Fig 1A–1D for each group ., As predicted , rats with the 120-s ITI tended to sign track more , whereas rats with the 60-s ITI tended to goal track more ., Across all behavioral indices ( i . e . , response bias , probability , latency , PCA ) , the mean distributions were positive ( biased toward sign tracking ) and significant from sessions for rats in the 120-s ITI group ( Fig 1A–1D , left; Wilcoxon; μ’s > 0 . 17 , p < 0 . 05 ) ., Opposite trends were observed in the 60-s ITI group in that all distributions were negatively shifted from zero ( Fig 1A–1D , right; Wilcoxon; response bias: μ = −0 . 06 , p = 0 . 06; lever probability: μ = −0 . 03 , p = 0 . 58; PCA index: μ = −0 . 11 , p = 0 . 097 ) ; however , only the shift in the latency difference distribution reached significance ( Fig 1C , right; Wilcoxon; μ = −0 . 10; p < 0 . 05 ) ., Direct comparisons between 60-s and 120-s ITI groups produced significant differences across all four measures ( Wilcoxons; p < 0 . 01 ) ., Thus , we conclude that lengthening the ITI increased sign-tracking behavior , as predicted by the STGT model 1 , 13 ., Notably , the degree of sign/goal tracking within the 60-s ITI group was highly dependent on when behavior was examined during the 8-s CS period ., This is illustrated in Fig 1G and Fig 1H , which show percent beam breaks in the food cup ( solid lines ) and lever pressing ( dashed lines ) over the time of the trial ., Consistent with the ratio analysis described above ( Fig 1A–1D ) , rats in the 120-s ITI group ( red ) showed sustained pressing ( red dashed ) that started shortly after lever extension and persisted throughout the 8-s CS period , while showing no increase in food cup entries ( red solid ) after CS presentation ( Fig 1G , red solid versus dashed ) ., Although it is clear that rats in the 120-s ITI group sign track more than goal track during the CS period , the relationship between lever pressing and food cup entry was far more dynamic during sessions with 60-s ITIs ( Fig 1G; blue ) ., During 60-s ITI sessions , rats would briefly enter the food cup for approximately 2 s immediately upon CS presentation ( Fig 1G , solid blue ) , before engaging with the lever ( Fig 1G , dashed blue ) ., As a result , lever pressing was delayed in the 60-s ITI group relative to the 120-s ITI group ( Fig 1G and 1H; blue versus red dashed ) ., This suggests that the goal-tracking tendencies described above during the entire 8-s CS period were largely due to the distribution of behaviors observed early in the CS period ., To quantify this observation , we recomputed the PCA index using either the first or the last 4 s of the 8-s CS period ., For the 120-s ITI group , the PCA index was significantly shifted in the positive direction during both the first and last 4 s of the cue period ( i . e . , more sign tracking; Fig 1E and 1F , left; Wilcoxon; μ’s > 0 . 16; p < 0 . 05 ) ., For the 60-s ITI group , the PCA index was significantly shifted in the negative direction during the first 4 s ( i . e . , more goal tracking; Fig 1E , right; Wilcoxon; μ = −0 . 16; p < 0 . 05 ) but not significantly shifted during the last 4 s ( Fig 1F , Wilcoxon; μ = 0 . 01; p = 0 . 81 ) ., Interestingly , this part of the results goes beyond the STGT model , which simplifies time by considering a single behavior/action during that period ., To further demonstrate sign- and goal-tracking tendencies over the 8-s cue period and the differences between groups , we simply subtracted 60-s ITI lever pressing and food cup entries from 120-s ITI lever pressing ( Fig 1I; orange ) and food cup entries ( Fig 1I; green ) , respectively ., Shortly after cue onset , the green line representing the difference between 120-s and 60-s ITI food cup entries dropped significantly below zero ., Throughout the cue period ( 8 s ) , there were more contacts with the food cup in sessions with a 60-s ITI compared with the 120-s ITI group ( green tick marks represent differences between 120-s and 60-s ITI across sliding 100-ms bins; t test; p < 0 . 05 ) ., For lever pressing ( orange ) , values were constantly higher shortly after the cue for the first half of the cue period ( orange tick marks represent differences between 120-s and 60-s ITI across sliding 100-ms bins; t test; p < 0 . 05 ) , indicating that there were more contacts with the lever in sessions with a 120-s ITI compared with the 60-s ITI group early in the cue period ., The behavioral data described above globally support model predictions that increasing and decreasing the ITI would produce more and less sign tracking , respectively ., Nevertheless , they also pave the way for improvements of the model by showing a rich temporal dynamic of behavior during the trial , rather than the single behavioral response per trial simulated in the model ., By plotting lever presses and food cup entries over time , we see that sometimes rats initially go to the lever and then go to the food cup , or vice versa ., In contrast , the model was designed to account only for the initial action performed by rats ., This was sufficient to account for the main results of the present study ., Nevertheless , it would be interesting to extend the model to enable it to account for different decisions made sequentially by the same animal during a given trial ., Next , we tested the prediction that longer ITIs would elevate DA release to the US , while shorter ITIs would reduce DA release to the US ., The average DA release over all sessions for the 60-s and 120-s groups is shown in Fig 2A ., Rats in the 120-s ITI group exhibited significantly higher DA release to the CS and the US relative to rats in the 60-s ITI group ( CS t test: t = 2 . 99 , df = 178 , p < 0 . 05; US t test: t = 3 . 07 , df = 178 , p < 0 . 05 ) ., In the 120-s ITI group , DA release to both the CS and the US was significantly higher than baseline ( CS t test: t = 14 . 77 , df = 119 , p < 0 . 05; US t test: t = 4 . 79 , df = 119 , p < 0 . 05 ) ; however , in the 60-s ITI group , this was only true during CS presentation ( t test: t = 7 . 34 , df = 59 , p < 0 . 05 ) ; DA release at the US was not different than baseline ( t test: t = 0 . 99 , df = 59 , p = 0 . 33 ) ., Similar results were obtained when averaging across sessions within each rat and then averaging across rats ( Fig 2B ) ; DA release was higher during the CS and US for rats in the 120-s ITI group ( CS t test: t = 1 . 87 , df = 17 , p < 0 . 05; US t test: t = 1 . 83 , df = 17 , p < 0 . 05 ) and was higher than baseline for both periods ( CS t test: t = 6 . 15 , df = 11 , p < 0 . 05; US t test: t = 2 . 16 , df = 11 , p < 0 . 05 ) , whereas DA release was only significantly higher during the CS period for rats in the 60-s ITI group ( CS t test: t = 6 . 68 , df = 6 , p < 0 . 05; US t test: t = 0 . 70 , df = 6 , p = 0 . 26 ) ., These results are in line with the STGT model , which predicted that reducing ITI duration would prevent the downward revision of the food cup value and hence would permit the high predictive value associated with the food cup to produce a DA response at CS but not US , consistent with the DA RPE hypothesis 9 ., Conversely and also consistent with model predictions , DA release during sessions with the longer ITI was significantly higher during US delivery because there were more positive RPEs , which may result from the positive surprise associated with being rewarded in a food cup whose value has been more strongly decreased during multiple visits to the food cup during long ITIs ., Nevertheless , at the CS time , the increased DA burst at CS indicates an even more complex process that goes beyond model predictions ., All of this suggests that DA release should be positively correlated with the time spent breaking the beam in the food cup during the ITI ., To test this hypothesis , we computed how much time was spent in the food cup during the ITI for each session ., This was done by determining the total number of beam breaks within each ITI ( 10-ms resolution ) and then averaging over trials to determine each session mean ., Importantly , the ITI time did not vary across sessions within each group , and the analysis was performed separately for the two groups ( 60-s group and 120-s group ) ., Thus , any correlation between DA and food cup interaction time during the ITI cannot reflect a correlation between DA and ITI time ., As expected , rats in the 120-s ITI group spent significantly more time in the food cup than did rats in the 60-s ITI group ( 120-s ITI group = 15 . 1 s; 60-s ITI group = 6 . 8 s; t test: t = 4 . 91 , df = 178 , p < 0 . 05 ) ., For both groups , there was a significant positive correlation between average time spent in the food cup during the ITI and DA release during the reward period ( Fig 2C , 120-s ITI: r2 = 0 . 12 , p < 0 . 05; Fig 2D , 60-s ITI: r2 = 0 . 08 , p < 0 . 05 ) ., During the cue period for the 120-s ITI group , but not the 60-s ITI group , there was also positive correlation ( Fig 2E , 120-s ITI: r2 = 0 . 04 , p < 0 . 05; Fig 2F , 60-s ITI: r2 = 0 . 01 , p = 0 . 36 ) ., Finally , when examining with data collapsed across both groups , there was a significant positive correlation during both cue and reward epochs ( Cue: r2 = 0 . 05 , p < 0 . 05; Reward: r2 = 0 . 14 , p < 0 . 05 ) ., Thus , we conclude that DA release to the CS and US tended to be higher the longer rats visited the food cup during the ITI ., In the analysis above , we averaged DA release and behavior from all recording sessions ., Next , we asked how behavior and DA release patterns evolved with training ., As a first step to addressing this issue , we recomputed the PCA analysis for the first and last 5 d of training ., For the 60-s ITI group , the PCA index distribution was significantly shifted in the negative direction ( i . e . , goal tracking ) during the first five sessions ( Wilcoxon; μ = −0 . 38 , p < 0 . 05 ) but not in the last five sessions ( Wilcoxon; μ = 0 . 15 , p = 0 . 07 ) ., Thus , early in training , rats with the 60-s ITI exhibited goal tracking more than sign tracking but did not fully transition to sign tracking , at least when we averaged over the last five sessions ., For the 120-s ITI group , the PCA index was significantly shifted in the positive direction ( i . e . , sign tracking ) during the last five sessions ( Wilcoxon; μ = 0 . 28 , p < 0 . 05 ) but was not during the first five sessions ( Wilcoxon; μ = 0 . 10 , p = 0 . 11 ) ., Thus , when the ITI was long ( 120 s ) , rats sign and goal tracked in roughly equal proportions during the first five sessions but tended to sign track significantly more during later sessions ., To more accurately pinpoint when during training rats in the 120-s group shift toward sign tracking , we examined the four distributions individually for each session ., Sign tracking became apparent during session 4 , when the latency and lever probability distributions first became significant ( Wilcoxon; latency: μ = 0 . 28 , p < 0 . 05; lever probability: μ = 0 . 40 , p < 0 . 05 ) ., To visualize changes in behavior and DA release that occurred before and after session 4 , we plotted food cup beam breaks , lever pressing , and DA release averaged across the first 3 d of training and across days 4–10 ( Fig 3; for visualization of behavior during each of the 10 sessions , please see S4 Fig ) ., Consistent with the distributions of behavioral indices described above , the 120-s ITI group showed roughly equal food cup entries and lever pressing during the CS period in the first 3 d of training ( Fig 3A , thin pink solid versus thin pink dashed ) , whereas later in training ( days 4–10; red ) , there was a strong preference for the lever ( Fig 3A; thick red dashed versus thick red solid ) ., Indeed , the distribution of PCA indices averaged during days 4–10 were significantly shifted in the positive direction ( Wilcoxon; μ = 0 . 27 , p < 0 . 05 ) ., These results suggest that in sessions in which the ITI was set at 120 s , sign-tracking tendencies developed relatively quickly during the first several recording sessions ( Fig 3A and 3C ) ., This is consistent with the STGT model , which predicted that increasing the ITI duration would increase the global tendency to sign track within the population and would thus speed up the acquisition of lever pressing behavior 1 , 13 ., In contrast , the model also predicted that reducing the ITI duration would increase the global tendency to goal track and would thus slow down the acquisition of lever pressing behavior ., Interestingly , the behavior of the 60-s ITI group was far more complicated than behavior of the 120-s group , with changes in goal and sign tracking occurring over training and CS presentation time ., Early in training , rats in the 60-s ITI group clearly visited the food cup ( Fig 3B , solid turquoise ) more than they pressed the lever ( Fig 3B , dashed turquoise ) ; food cup entries increased shortly after presentation of the CS and continued throughout the CS period ( Fig 3B , solid turquoise ) ., During later sessions ( i . e . , 4–10 ) , rats in the 60-s ITI group still entered the food cup upon CS presentation—which corresponds to the goal-tracking behavior predicted by the model in this case—but this only lasted about 2 s , at which point they transitioned to the lever ( Fig 3B and 3D ) ., In sessions 4–10 , none of the distributions of behavioral indices were significantly shifted from zero when examining the CS period as a whole ( Wilcoxons; Response bias: μ = 0 . 27 , p = 0 . 83; Latency: μ = −0 . 05 , p = 0 . 13; Probability: μ = 0 . 08 , p = 0 . 16; PCA: μ = 0 . 02 , p = 0 . 82 ) or during the first half of the CS period ( Response bias: μ = −0 . 11 , p = 0 . 027; Probability: μ = −0 . 04 , p = 0 . 18; PCA: μ = −0 . 07 , p = 0 . 25 ) ; however , when examining the last 4 s of the CS period , distributions were significantly shifted in the positive direction ( Wilcoxons; Response bias: μ = 0 . 32 , p < 0 . 05; Probability: μ = 0 . 28 , p < 0 . 05; PCA: μ = 0 . 24 , p < 0 . 05 ) ., Together , this suggests that rats in the 60-s groups were largely goal tracking early in training and that over the course of training , goal-tracking tendencies did not disappear but became focused to early portions of the CS period , while sign-tracking behavior developed toward the end of the CS period , later in training ( Fig 3B and 3D; S4 Fig ) ., Interestingly , these results go again beyond the computational model and suggest that it should be extended to account for within-trial behavioral variations ., Behavioral analyses clearly demonstrate that manipulation of the ITI impacts sign- and goal-tracking behavior and that both groups learned that the CS predicted reward ( Fig 3; S4 Fig ) ., Next , we determined how DA patterns changed during training ., Fig 3E and 3F illustrate DA release averaged across the first 3 d and days 4–10 of sessions with 120-s and 60-s ITIs , respectively , and DA release for each session is plotted in Fig 3G and 3H ., As shown previously , both groups started with modest DA release to both the CS and US during the first session ( Fig 3G and 3H; trial 1 ) ., For the 120-s ITI group , DA release was significantly higher to CS presentation later ( red ) compared to earlier ( pink ) in learning ( Fig 3E; t test: t = 2 . 51 , df = 119 , p < 0 . 05 ) ., DA release during US delivery did not significantly differ between early and late phases of training ( t test: t = 1 . 27 , df = 119 , p = 0 . 21 ) ., Hence , similarly to the sign trackers in the original study of Flagel and colleagues ( 2011 ) , the increase of DA response to the CS is consistent with the RPE hypothesis ., The difference is that here , the increase in the time available to down-regulate the value associated with the food cup during the ITI may have resulted in a remaining positive surprise at the time of reward delivery , hence preventing the progressive decrease of response to the US across training , in accordance with the model predictions ., In the 60-s ITI group ( Fig 3F and 3H ) , DA release to the US was initially high during the first 3 d ( turquoise ) but declined during days 4–10 ( blue ) ., Directly comparing DA release during the first 3 d with the remaining days revealed significant differences during the US period ( t test: t = 1 . 14 , df = 59 , p < 0 . 05 ) but not the CS period ( t test: t = 0 . 08 , df = 59 , p = 0 . 93 ) ., As a consequence , their post-training DA pattern—with a high response to the CS but no response to the US ( Fig 3F , blue ) —now resembles the traditional RPE pattern ( i . e . , high CS DA and low US DA after learning ) ., This is a clear demonstration that the DA RPE signal can be observed in goal trackers with a manipulation of the ITI , as predicted by the STGT model ., In a final analysis , we examined DA patterns during pure sign and goal tracking within each ITI group ., For this analysis , we examined only sessions during which either the lever was pressed or the food cup was entered during the cue period ., As shown previously 4 , phasic DA responses were apparent during both the CS and US during sessions with goal tracking ( Fig 3I and 3J , GT = orange ) ., In addition to replicating previous results , the figure also illustrates modulation of the DA pattern in line with model predictions ., Specifically , it shows that the DA response to the US was higher in the 120-s group than in the 60-s group during both sign- and goal-tracking sessions ( sign-tracking: t test , t = 3 . 66 , df = 25 , p < 0 . 05; goal-tracking: t test , t = 1 . 44 , df = 29 , p = 0 . 16 ) and that the DA response to the US was significantly lower than the DA response to the CS in GTs of the 60-s group ( t = 3 . 87 , df = 17 , p < 0 . 05 ) , suggesting that even though there is still a DA response to the US , shortening the ITI reduced the US-evoked DA response compared with what has been previously reported 4 ., The results reported here support the STGT model’s predictions that manipulating the ITI would impact the proportion of sign-tracking ( STs ) and goal-tracking ( GTs ) behaviors as well as DA release ., It predicted that shortening the ITI would result in fewer negative revisions of the food cup value and reduce the US DA burst ., It also predicted that the resulting higher food cup value would lead to an increase in the tendency to GT across sessions 1 , 13 , which it did ., The model also predicted that lengthening the ITI would have the opposite effect ., We found that there were significantly more food cup entries during the ITI for the 120-s ITI group and that they showed an increased tendency to sign track ., Furthermore , we show that the time spent in the food cup during the ITI was positively correlated with the amplitude of the CS and US DA bursts for the 120-s ITI group , which is consistent with the hypothesis that lengthening the ITI to allow for more time to decrease the value of the food cup would result in stronger positive RPEs during the trial ., Consistent with the model , we claim that increased sign tracking and DA release result from the additional time spent in the food cup during the ITI ., Indeed , these were positively correlated ., Importantly , this impact of ITI manipulations had not been predicted by other computational models of sign trackers and goal trackers 11 , 14 ., However , several alternative explanations should be considered , which may have also contributed to observed changes in behavior and DA release ., For example , it has been shown that rewards delivered after longer delays yield higher DA responses to the US 15–17 and that uncertain reward increases sign tracking 18 ., Although the reward was highly predictable in our study ( i . e . , always delivered 8 s after cue onset ) , it is possible that uncertainty associated with US delivery impacted behavior and DA release ., Notably , it is likely that these factors are intertwined in that manipulating delays and certainty impact the number of visits to the food cup that are not rewarded , thus leading to a negative revision of the food cup , as predicted by the model ., Future work that modifies food cup entries without manipulating ITI length and rewards uncertainty is necessary to determine the unique contributions that these factors play in goal-/sign-tracking behavior and associated DA release ., Another explanation for increased sign tracking and DA release in the rats in the 120-s ITI group is the possibility that they learned faster than rats in the 60-s ITI group because of differing ratios between US presentations and the interval between the CS and US in that , the shorter the CS-US interval relative to the ITI , the faster the learning 19 ., In the context of our study it is difficult to determine which group learned faster ., Although rats in the 120-s ITI group did lever press more often early in training , rats in the 60-s ITI group made more anticipatory food cup entries during the cue period prior to reward delivery ., Furthermore , both food cup entries and lever pressing were present in the first behavioral session ( S4 Fig ) ., Thus , both groups appear to learn the CS-US relationship at similar speeds , but it is just that the behavior readout of learning differs across groups , making it difficult to determine which group learned the association faster ., In our opinion , our results suggest that rats in both groups learned at similar rates , much like sign and goal trackers do; however , future experiments and iterations of the model are necessary to determine what role the US-US/CS-US ratio plays in sign/goal tracking and corresponding DA release ., Standard RL 20 is a widely used normative framework for modelling learning experiments 21 , 22 ., To account for a variety of observations suggesting that multiple valuation processes coexist within the brain , two main classes of models have been proposed: MB and MF models 23 , 24 ., MB systems employ an explicit , although approximate , internal model of the consequences of actions , which makes it possible to evaluate situations by forward inference ., Such systems best explain goal-directed behaviors and rapid adaptation to novel or changing environments 25–28 ., In contrast , MF systems do not rely on internal models but directly associate stored ( cached ) values with actions or states based on experience , such that higher valued situations are favored ., Such systems best explain habits and persistent behaviors 28–30 ., Learning in MF systems relies on a computed reinforcement signal , the RPE ( actual minus predicted reward ) ., This signal has been shown to correlate with the phasic response of midbrain DA neurons that increase and decrease firing to unexpected appetitive and aversive events , respectively 9 , 31 ., Recent work by Flagel and colleagues 4 has questioned the validity of classical MF RL methods in Pavlovian conditioning experiments ., Their autoshaping procedure reported in that article was nearly identical to the one presented here in that a retractable-lever CS was presented for 8 s , followed immediately by delivery of a food pellet into an adjacent food cup ., The only major difference was that the length of the ITI in their study was 90 s ., In their study , they showed that in STs , phasic DA release in the NAc matched RPE signaling ., That is , the DA burst to reward that was present early in learning transferred to the cue after learning ., They also showed that DA transmission was necessary for the acquisition of sign tracking ., In contrast , despite the fact that GTs acquired a Pavlovian conditioned approach response , this was not accompanied by the expected RPE-like DA signal , nor was the acquisition of the goal-tracking conditioned response blocked by administration of a DA antagonist ( see also Danna and Elmer 10 ) ., To account for these and other results , Khamassi and colleagues 1 proposed a new computational model—the STGT model—that explains a large set of behavioral , physiological , and pharmacological data obtained from studies on individual variation in Pavlovian conditioned approach 2–8 ., Importantly , the model can reproduce previous experimental data by postulating that both MF and MB learning mechanisms occur during behavior , with simulated interindividual variability resulting from a different weight associated with the contribution of each system ., The model accounts for the full spectrum of observed behaviors ranging from one extreme—from sign tracking associated with a small contribution of the MB system in the model—to the other—goal tracking associated with a high contribution of the MB system in the model 12 ., Above all , by allowing the MF system to learn different values associated with different stimuli , depending on the level of interaction with those stimuli , the model potentially explains why the lever CS and the food cup might acquire different motivational values in different individuals , even when they undergo the same training in the same task 26 ., The STGT model explains why the RPE-like dopaminergic response was observed in STs but not GTs—the proposition being that GTs would focus on the reward-predictive value of the food cup , which would have been down-regulated during the ITI ., Furthermore , the STGT explains why inactivating DA in the core of the nucleus accumbens or in the entire brain results in blocking specific components and not others ., Here , the model proposes that learning in GTs relies more heavily on the DA-independent MB system , and thus DA blockade would not impair learning in these individuals 4 , 8 ., More importantly , the model has led to a series of new experimentally testable predictions that assess and strengthen the proposed computational theory and allow for a better understanding of the DA-dependent and DA-independent mechanisms underlying interindividual differences in learning 1 , 13 ., The key computational mechanism in the model is that both the approach and the consumption-like engagement observed in sign trackers ( STs ) on the lever and in goal trackers ( GTs ) on the food cup result from the acquisition of incentive salience by these reward-predicting stimuli ., Acquired incentive salience is stimulus specific: stimuli most predictive of reward will be the most “wanted” by the animal ., The MF system attributes accumulating salience to the lever or the food cup as a function of the simulated DA phasic signals ., In the model simulations , because the food cup is accessible but not rewarding during the ITI , a simulated negative DA error signal occurs each time the animal visits the food cup and does not find a reward ., The food cup therefore acquires less incentive salience compared with the lever , which is only presented prior to reward delivery ., In simulated STs , behavior is highly subject to incentive salience because of a higher weight attributed to the MF system than to the MB
Recent computational models of sign tracking ( ST ) and goal tracking ( GT ) have accounted for observations that dopamine ( DA ) is not necessary for all forms of learning and have provided a set of predictions to further their validity ., Among these , a central prediction is that manipulating the intertrial interval ( ITI ) during autoshaping should change the relative ST-GT proportion as well as DA phasic responses ., Here , we tested these predictions and found that lengthening the ITI increased ST , i . e . , behavioral engagement with conditioned stimuli ( CS ) and cue-induced phasic DA release ., Importantly , DA release was also present at the time of reward delivery , even after learning , and DA release was correlated with time spent in the food cup during the ITI ., During conditioning with shorter ITIs , GT was prominent ( i . e . , engagement with food cup ) , and DA release responded to the CS while being absent at the time of reward delivery after learning ., Hence , shorter ITIs restored the classical DA reward prediction error ( RPE ) pattern ., These results validate the computational hypotheses , opening new perspectives on the understanding of individual differences in Pavlovian conditioning and DA signaling .
In classical or Pavlovian conditioning , subjects learn to associate a previously neutral stimulus ( called “conditioned” stimulus; for example , a bell ) with a biologically potent stimulus ( called “unconditioned” stimulus; for example , a food reward ) ., In some animals , the incentive salience of the conditioned stimuli is so strong that the conditioned response is to engage the conditioned stimuli instead of immediately approaching the food cup , where the predicted food will be delivered ., These animals are referred to as “sign trackers . ”, Other animals , referred to as “goal trackers , ” proceed directly to the food cup upon presentation of the conditioned stimulus to obtain reward ., Understanding the mechanisms by which these divergent behaviors develop under identical environmental conditions will provide powerful insight into the neurobiological substrates underlying learning ., Here , we test predictions made by a recent computational model that accounts for a large set of studies examining goal-/sign-tracking behavior and the role that dopamine plays in learning ., We show that increasing the duration of the time between trials leads more to the development of a sign-tracking response and to the release of dopamine in the nucleus accumbens ., During conditioning with shorter intertrial intervals , goal tracking was more prominent , and dopamine was released upon presentation of the conditioned stimulus but not during the time of reward delivery after training ., Thus , shorter intertrial intervals restored the classical dopamine reward prediction error pattern ., Our results validate the computational hypothesis and open the door for understanding individual differences to classical conditioning .
learning, medicine and health sciences, neurochemistry, chemical compounds, classical conditioning, vertebrates, social sciences, conditioned response, neuroscience, animals, mammals, learning and memory, organic compounds, hormones, animal models, surgical and invasive medical procedures, model organisms, cognitive psychology, mathematics, functional electrical stimulation, probability distribution, membrane electrophysiology, experimental organism systems, amines, neurotransmitters, bioassays and physiological analysis, catecholamines, dopamine, research and analysis methods, animal studies, behavior, chemistry, electrophysiological techniques, short reports, probability theory, biochemistry, behavioral conditioning, rodents, psychology, eukaryota, electrode recording, organic chemistry, biogenic amines, biology and life sciences, physical sciences, cognitive science, amniotes, organisms, rats