insight

review

articles

Computational

roles

for

dopamine

in

behavioural

control

P.

Read

Steven

E.

Jonathan

D.

Department

of

Neuroscience

and

Department

of

Psychiatry

and

Behavioral

Sciences

Baylor

College

of

Medicine

Baylor

Plaza

Houston

Texas

USA

-LRB-

e-mail

read@bcm.tmc.edu

-RRB-

Harvard

University

Cambridge

Massachusetts

USA

-LRB-

e-mail

seh@harvard.edu

-RRB-

Department

of

Psychiatry

University

of

Pittsburgh

and

of

Psychology

Center

for

the

Study

of

Brain

Mind

Behavior

Green

Hall

Princeton

University

Princeton

New

Jersey

USA

-LRB-

e-mail

jdc@princeton.edu

-RRB-

Neuromodulators

such

as

dopamine

have

a

central

role

in

cognitive

disorders

In

the

past

decade

biological

findings

on

dopamine

function

have

been

infused

with

concepts

taken

from

computational

theories

of

reinforcement

learning

These

more

abstract

approaches

have

now

been

applied

to

describe

the

biological

algorithms

at

play

in

our

brains

when

we

form

value

judgements

and

make

choices

The

application

of

such

quantitative

models

has

opened

up

new

fields

ripe

for

attack

by

young

synthesizers

and

theoreticians

he

concept

of

behavioural

control

is

intimately

tied

to

the

valuation

of

resources

and

choices

For

example

a

creature

that

moves

left

instead

of

right

may

forgoe

the

food

and

other

resources

that

it

could

have

obtained

had

it

chosen

right

Such

stark

yet

simple

economic

realities

select

for

creatures

that

evaluate

the

world

quickly

and

choose

appropriate

behaviour

based

on

those

valuations

From

the

point

of

view

of

selection

the

most

effective

valuations

are

those

that

improve

reproductive

success

This

prescription

for

valuation

yields

a

formula

for

desires

or

goals

an

organism

should

desire

those

things

deemed

most

valuable

to

it

All

mobile

organisms

possess

such

discriminatory

capacities

and

can

rank

numerous

dimensions

in

their

world

along

axes

that

extend

from

good

to

bad

A

kind

of

facile

biological

wisdom

is

built

into

these

simple

observations

and

we

should

expect

valuation

mechanisms

to

be

built

into

our

nervous

systems

at

every

level

from

the

single

neuron

to

the

decision

algorithms

used

in

complex

social

settings

These

ideas

have

recently

been

upgraded

from

provocative

biological

musings

to

real

computational

models

of

how

the

nervous

system

sets

goals

computes

values

of

particular

resources

or

options

and

uses

both

to

guide

sequences

of

behavioural

choices

Such

models

have

cast

as

important

players

our

midbrain

's

dopamine

neurons

whose

actions

define

rewards

our

goals

or

desires

that

should

be

sought

These

neurons

have

a

central

role

in

guiding

our

behaviour

and

thoughts

They

are

hijacked

by

every

addictive

drug

they

malfunction

in

mental

illness

and

they

are

lost

in

dramatically

impairing

illnesses

such

as

Parkinson

's

disease

If

dopamine

systems

are

overstimulated

we

may

hear

voices

experience

elaborate

bizarre

cognitive

distortions

or

engage

excessively

in

dangerous

goal-directed

behaviour

Dopamine

function

is

also

central

to

the

way

that

we

value

our

world

including

the

way

that

we

value

money

and

other

human

beings

The

full

story

of

behavioural

control

requires

vastly

more

than

simple

models

of

dopaminergic

function

But

here

we

show

how

one

branch

of

computational

theory

reinforcement

learning

has

informed

both

the

design

and

interpretation

of

experiments

that

probe

how

the

dopamine

system

influences

sequences

of

choices

made

about

rewards

These

models

are

maturing

rapidly

and

may

even

guide

our

understanding

of

other

neuromodulatory

systems

in

the

brain

although

such

applications

are

still

in

their

infancy

Nature

Publishing

Group

T

Reinforcement

signals

define

an

agent

's

goals

Reinforcement

learning

theories

seek

to

explain

how

organisms

learn

to

organize

their

behaviour

under

the

influence

of

Reward

is

an

old

psychological

term

defined

by

Merriam

Webster

's

dictionary

as

a

stimulus

administered

to

an

organism

following

a

correct

or

desired

response

that

increases

the

probability

of

occurrence

of

the

response

Here

we

show

that

current

theories

of

reinforcement

learning

provide

a

formal

framework

for

connecting

the

physiological

actions

of

specific

neuromodulatory

systems

to

behavioural

control

We

focus

on

dopaminergic

systems

primarily

because

they

have

been

most

extensively

modelled

and

because

they

play

a

major

role

in

decision-making

motor

output

executive

control

and

reward-dependent

We

show

how

the

dopaminergic

models

provide

a

way

to

understand

neuroimaging

experiments

on

reward

expectancy

and

cognitive

control

in

human

subjects

Finally

we

suggest

that

this

same

class

of

model

has

matured

sufficiently

for

it

to

be

used

to

address

important

disturbances

in

neuromodulation

associated

with

many

psychiatric

disorders

Despite

its

name

reinforcement

learning

is

not

simply

a

modern

recapitulation

of

stimulus

response

learning

familiar

from

the

classical

and

instrumental

conditioning

Traditional

stimulus

response

models

focused

on

how

direct

associations

can

be

learned

between

stimuli

and

responses

overlooking

the

possibility

that

numerous

internal

states

intervene

between

the

stimulus

and

its

associated

response

However

animals

clearly

have

covert

internal

states

that

affect

overt

measurable

behaviour

Reinforcement

learning

theory

explicitly

models

such

intervening

states

assumes

that

some

are

more

desirable

than

others

and

asks

how

do

animals

learn

to

achieve

desired

states

and

avoid

undesirable

ones

as

efficiently

as

possible

The

answer

to

this

question

shows

how

reinforcement

signals

define

an

agent

's

goals

For

simplicity

we

focus

only

on

rewards

However

the

same

story

can

be

told

using

negative

reinforcers

-LRB-

punishments

-RRB-

We

refer

to

the

state

engendered

by

a

reward

as

a

goal

Goals

can

exist

at

numerous

levels

and

direct

behaviour

over

many

timescales

Goals

for

humans

range

from

the

most

basic

-LRB-

for

example

procuring

something

to

eat

in

the

next

minute

-RRB-

to

the

most

abstract

and

complex

-LRB-

such

as

planning

a

career

-RRB-

In

reinforcement

learning

it

is

assumed

that

the

fundamental

goal

of

the

agent

-LRB-

learner

-RRB-

is

to

learn

to

take

actions

that

are

most

likely

to

lead

to

the

greatest

accrual

of

rewards

in

the

future

This

goal

is

achieved

under

the

guidance

of

simple

scalar

quantities

called

reinforcement

signals

These

signals

NATURE

VOL

OCTOBER

insight

review

articles

reward

and

predicted

reward

instead

it

incorporates

information

about

the

next

prediction

made

by

the

reward-prediction

In

words

current

TD

error

current

reward

next

prediction

current

prediction

Here

the

words

current

and

next

refer

respectively

to

the

present

state

and

to

the

subsequent

state

of

the

learner

is

a

factor

between

and

that

weights

the

relative

influence

of

the

next

prediction

By

using

this

reward-prediction

error

to

refine

predictions

of

reward

for

each

state

the

system

can

improve

its

estimation

of

the

value

of

each

state

and

improve

its

policy

function

's

ability

to

choose

actions

that

lead

to

more

reward

The

reward-prediction-error

hypothesis

Figure

TD

prediction-error

signal

encoded

in

dopamine

neuron

firing

Electrophysiological

recordings

from

a

single

dopamine

neuron

in

a

monkey

during

reward-dependent

discrimination

task

The

animal

presses

a

key

two

pictures

are

presented

the

animal

releases

the

key

and

hits

the

lever

under

the

rewarded

picture

If

a

correct

choice

is

made

juice

is

delivered

after

a

fixed

delay

Juice

delivery

is

marked

by

vertical

bars

neuron

spikes

by

dots

Early

on

the

juice

delivery

causes

a

burst

response

-LRB-

top

blue

arrowhead

-RRB-

This

surprise

response

diminishes

to

zero

by

the

end

of

learning

-LRB-

bottom

blue

arrowhead

-RRB-

A

catch

trial

using

a

-LRB-

surprising

-RRB-

delayed

reward

time

exposes

the

typical

pause

-LRB-

red

arrowhead

-RRB-

and

burst

-LRB-

top

green

arrowhead

-RRB-

response

The

pause

signals

that

things

are

worse

than

expected

and

the

burst

signals

that

things

are

better

than

expected

In

the

second

catch

trial

the

reward

is

again

surprising

but

early

rather

than

late

The

burst

response

for

the

new

delivery

time

is

apparent

-LRB-

lower

green

arrowhead

-RRB-

but

the

pause

response

is

less

certain

-LRB-

red

question

mark

-RRB-

Adapted

from

ref

Over

the

past

decade

experimental

work

by

Wolfram

Schultz

and

colleagues

has

shown

that

dopaminergic

neurons

of

the

ventral

tegmental

area

and

substantia

nigra

show

phasic

changes

in

spike

activity

that

correlate

with

the

history

of

reward

It

was

proposed

that

these

phasic

activity

changes

encode

a

prediction

error

about

summed

future

reward

-LRB-

as

described

above

-RRB-

this

hypothesis

has

been

tested

successfully

against

a

range

of

physiological

The

pause

and

burst

responses

of

dopamine

neurons

that

support

a

reward-prediction-error

hypothesis

are

shown

in

Fig.

The

bursts

signal

a

positive

reward-prediction

error

-LRB-

things

are

better

than

expected

-RRB-

and

the

pauses

signal

a

negative

prediction

error

-LRB-

things

are

worse

than

expected

-RRB-

Activity

that

remains

close

to

the

baseline

signals

that

things

are

just

as

expected

However

this

verbal

interpretation

of

dopaminergic

activity

belies

the

sophistication

of

the

underlying

neural

-LRB-

Box

-RRB-

Value

binding

and

incentive

salience

serve

to

criticize

specific

actions

or

contemplated

actions

with

respect

to

how

effectively

they

serve

the

agent

's

goals

In

reinforcement

learning

one

common

goal

is

the

maximization

of

total

future

Every

reinforcement

learning

system

possesses

three

explicitly

implemented

components

-LRB-

-RRB-

a

reinforcement

signal

that

assigns

a

numerical

quantity

to

every

state

of

the

agent

Reinforcement

signals

can

be

negative

or

positive

They

define

the

agent

's

immediate

goals

by

reporting

on

what

is

good

or

bad

right

now

-LRB-

-RRB-

a

stored

value

function

that

formalizes

the

idea

of

longer-term

judgments

by

assigning

a

value

to

the

current

state

of

the

agent

-LRB-

see

Box

-RRB-

-LRB-

-RRB-

a

policy

function

that

maps

the

agent

's

states

to

its

actions

Policies

are

typically

stochastic

they

assign

a

probability

to

each

possible

action

that

can

be

taken

from

the

current

state

with

the

probability

weighted

by

the

value

of

the

next

state

produced

by

that

action

A

more

concrete

description

reads

as

iterations

of

the

following

recipe

-LRB-

-RRB-

organism

is

in

state

X

and

receives

reward

information

-LRB-

-RRB-

organism

queries

stored

value

of

state

X

-LRB-

-RRB-

organism

updates

stored

value

of

state

X

based

on

current

reward

information

-LRB-

-RRB-

organism

selects

action

based

on

stored

policy

and

-LRB-

-RRB-

organism

transitions

to

state

Y

and

receives

reward

information

In

one

form

of

reinforcement

learning

called

temporal-difference

learning

a

critical

signal

is

the

reward-prediction

error

-LRB-

also

called

the

temporal-difference

or

TD

error

-RRB-

Unlike

the

well-known

psychological

learning

rule

proposed

by

Rescorla

and

in

this

error

function

is

not

simply

a

difference

between

the

received

NATURE

VOL

OCTOBER

We

have

presented

theoretical

evidence

that

phasic

bursts

and

pauses

in

midbrain

dopaminergic

activity

are

consistent

with

the

formal

construct

of

a

reward-prediction

error

used

by

reinforcement

learning

systems

-LRB-

Fig.

Box

-RRB-

This

interpretation

is

consistent

with

a

long

history

of

physiological

and

pharmacological

data

showing

that

dopamine

is

involved

in

appetitive

approach

and

is

a

key

component

in

the

pathologies

of

behavioural

control

associated

with

drug

One

finding

offered

as

a

challenge

to

the

models

discussed

so

far

is

that

antagonism

of

dopamine

receptors

does

not

change

the

appetitive

value

of

food

rewards

but

does

prevent

the

treated

animal

from

initiating

actions

that

allow

it

to

obtain

the

food

In

these

experiments

animals

treated

with

dopamine-receptor

blockers

are

virtually

unable

to

link

sequences

of

actions

to

obtain

a

food

reward

but

they

will

consume

the

same

amount

as

untreated

animals

if

they

are

moved

close

to

the

food

rewards

by

the

experimenter

-LRB-

Fig.

-RRB-

This

conclusion

also

holds

for

the

inhibition

of

dopamine

neuron

firing

by

gamma-aminobutyric

acid

-LRB-

GABA

-RRB-

injected

directly

into

the

ventral

tegmental

area

-LRB-

Fig.

-RRB-

These

data

suggest

that

interfering

with

dopamine

transmission

does

not

alter

the

internal

evaluation

of

rewards

but

simply

the

ability

to

act

on

those

valuations

Addressing

these

data

at

a

conceptual

level

Berridge

and

Robinson

have

proposed

that

dopamine

mediates

the

binding

between

the

hedonic

evaluation

of

stimuli

and

the

assignment

of

these

values

to

objects

or

They

call

this

idea

incentive

salience

Although

competing

psychological

explanations

differ

with

respect

to

the

specific

claims

of

incentive

they

all

agree

that

dopamine

release

and

binding

is

a

necessary

link

between

the

evaluation

of

potential

future

rewards

and

the

policy

-LRB-

sequence

of

actions

-RRB-

that

acquires

the

rewards

Here

we

refer

to

this

link

as

value

binding

and

distinguish

three

components

-LRB-

-RRB-

the

value

computation

-LRB-

-RRB-

the

link

to

a

policy

-LRB-

value

binding

-RRB-

and

-LRB-

-RRB-

execution

of

the

policy

Incentive

salience

and

actor

critic

models

There

is

a

class

of

reinforcement

learning

model

called

the

actor

critic

that

is

closely

related

to

the

Berridge

and

Robinson

model

for

the

role

of

dopamine

in

value

and

action

In

these

models

the

critic

carries

the

reward-prediction

error

associated

Nature

Publishing

Group

insight

review

articles

Box

Value

functions

and

prediction

errors

The

value

function

In

the

simplest

TD

models

of

dopamine

systems

the

reward-prediction

error

depends

on

a

value

function

that

equates

the

value

V

of

the

current

state

s

at

time

t

with

the

average

sum

of

future

rewards

received

up

until

the

end

of

a

learning

trial

V

-LRB-

st

-RRB-

average

sum

of

future

rewards

delivered

from

state

st

until

the

end

of

a

learning

trial

average

-LSB-

rt

rt

rt

r

-LRB-

trial

's

end

-RRB-

-RSB-

E

r

-LRB-

-RRB-

Trial

the

value

of

its

successor

state

s

t

Until

this

point

we

have

been

discussing

the

ideal

case

for

V.

However

as

indicated

above

V

can

not

be

known

exactly

in

the

real

world

Instead

an

estimate

V

of

V

must

be

formed

within

the

nervous

system

The

TD

algorithm

learns

an

approximation

V

of

the

value

function

V

It

uses

a

prediction-error

signal

-LRB-

t

-RRB-

prediction

error

-LRB-

t

-RRB-

E

-LSB-

rt

-RSB-

V

-LRB-

st

-RRB-

V

-LRB-

st

-RRB-

current

reward

next

prediction

current

prediction

-LRB-

-RRB-

-LRB-

-RRB-

This

TD

error

signal

reproduces

the

phasic

burst

and

pause

responses

measured

in

dopamine

neurons

recorded

in

alert

monkeys

during

learning

tasks

The

next

value

of

each

adaptable

weight

w

-LRB-

t

-RRB-

used

to

estimate

V

is

incremented

or

decremented

in

proportion

to

the

product

of

the

current

prediction

error

-LRB-

t

-RRB-

and

the

current

representation

s

-LRB-

t

-RRB-

of

the

stimulus

responsible

for

the

prediction

w

-LRB-

t

-RRB-

w

-LRB-

t

-RRB-

s

-LRB-

t

-RRB-

-LRB-

t

-RRB-

-LRB-

-RRB-

E

is

the

expected

value

operator

There

are

two

sources

of

randomness

over

which

the

above

averaging

occurs

First

the

rewards

in

a

trial

-LSB-

rt

rt

rt

r

-LRB-

trial

's

end

-RRB-

-RSB-

are

random

variables

indexed

by

the

time

t.

For

example

rt

is

a

sample

of

the

distribution

of

rewards

received

two

timesteps

into

the

trial

The

idea

is

that

the

animal

can

learn

the

average

value

of

these

rewards

by

repeating

learning

trials

and

by

revisiting

state

st

sufficiently

frequently

for

its

nervous

system

to

be

able

to

estimate

the

average

value

of

each

of

the

rewards

received

from

state

st

until

the

end

of

the

trial

The

second

source

of

randomness

is

the

probabilistic

transition

from

one

state

at

time

t

to

a

succeeding

state

s

t

at

a

later

time

t

The

value

function

stored

within

the

nervous

system

of

the

creature

provides

an

assessment

of

the

likely

future

rewards

for

each

state

of

the

creature

that

is

the

value

must

somehow

be

associated

with

the

state

However

as

written

in

equation

-LRB-

-RRB-

it

would

be

virtually

impossible

to

make

good

estimates

of

the

ideal

V

-LRB-

st

-RRB-

as

it

is

now

defined

This

is

because

the

creature

would

have

to

wait

until

all

rewards

were

received

within

a

trial

before

deciding

on

the

value

of

its

state

at

the

beginning

of

the

trial

By

that

time

it

is

too

late

for

such

a

computation

to

be

useful

This

problem

becomes

worse

in

real-world

settings

Fortunately

equation

-LRB-

-RRB-

provides

a

way

out

of

this

dilemma

because

it

obeys

a

recursion

relation

through

time

V

-LRB-

st

-RRB-

E

-LSB-

r

t

-RSB-

V

-LRB-

s

t

-RRB-

-LRB-

-RRB-

Here

is

a

learning

rate

Exponential

discounting

of

future

rewards

The

artificial

truncation

at

the

end

of

a

trial

-LRB-

equation

-LRB-

-RRB-

-RRB-

can

be

handled

theoretically

in

several

ways

One

popular

formalization

is

to

weight

the

near

future

more

than

the

distant

future

In

this

case

the

analogue

to

equation

-LRB-

-RRB-

takes

the

form

Vd

-LRB-

-LRB-

t

-RRB-

-RRB-

average

sum

of

discounted

future

rewards

average

-LSB-

-LRB-

t

-RRB-

-LRB-

t

-RRB-

-LRB-

t

-RRB-

-RSB-

for

E

t

t

r

-LRB-

-RRB-

Using

this

weighted

version

of

the

value

function

the

learning

episodes

for

a

creature

do

not

have

to

be

artificially

divided

into

trials

An

analogous

reward-prediction-error

signal

can

be

formed

and

used

in

the

same

manner

as

above

-LRB-

t

-RRB-

prediction

error

-LRB-

t

-RRB-

E

-LSB-

rt

-RSB-

V

-LRB-

st

-RRB-

V

-LRB-

st

-RRB-

current

reward

next

prediction

current

prediction

This

recursion

relation

shows

that

information

about

the

value

of

a

state

s

t

is

available

using

only

the

value

V

-LRB-

st

-RRB-

of

the

current

state

s

t

and

-LRB-

-RRB-

a

Value

binding

Dopamine

required

to

bind

value

to

action

Start

Intake

Running

Activity

b

Actor

critic

models

Value

function

Reward

function

V

-LRB-

s

-LRB-

t

-RRB-

-RRB-

V

-LRB-

s

-LRB-

t

-RRB-

-RRB-

r

-LRB-

t

-RRB-

Goal

Control

-LRB-

-RRB-

Block

dopamine

binding

in

NAc

Critic

-LRB-

t

-RRB-

Dopamine

neurons

-LRB-

VTA

-RRB-

Actor

Inhibit

dopamine

neuron

spiking

in

VTA

wi

-LRB-

t

-RRB-

Reward

learning

P

-LRB-

e

-LRB-

m

-LRB-

t

-RRB-

b

-RRB-

-RRB-

Action

selection

Figure

Equating

incentive

salience

with

the

actor

critic

model

a

Rats

are

trained

to

run

a

maze

to

acquire

sugary

water

If

dopaminergic

spiking

is

blocked

-LRB-

left

histograms

-RRB-

in

the

VTA

then

rats

will

generally

not

run

down

the

maze

to

get

a

reward

and

are

less

active

However

if

the

experimenter

moves

them

to

the

sugary

water

the

rats

drink

exactly

the

same

amount

as

untreated

rats

This

suggests

that

the

-LRB-

hedonic

-RRB-

value

of

the

sugary

water

has

been

computed

but

that

the

capacity

to

bind

this

value

to

actions

required

to

obtain

the

water

fails

to

function

The

same

effect

results

if

dopamine

's

interaction

with

its

receptor

is

blocked

in

an

important

downstream

target

of

dopamine

projections

-LRB-

right

histograms

-RRB-

Adapted

from

refs

and

b

Actor

critic

models

use

dopamine-encoded

prediction-error

signal

in

two

roles

-LRB-

-RRB-

to

learn

stimulus

reward

associations

and

-LRB-

-RRB-

to

assess

actions

or

contemplated

actions

-LRB-

notations

are

as

in

Box

-RRB-

Adapted

from

refs

NATURE

VOL

OCTOBER

Nature

Publishing

Group

insight

review

articles

anticipated

outcome

anticipated

outcome

anticipated

outcome

Time

-LRB-

s

-RRB-

Figure

Scaled

responses

to

a

monetary

reward

in

the

ventral

striatum

Action

is

required

to

receive

a

reward

The

haemodynamic

response

is

modulated

by

the

amount

of

money

received

In

both

cases

positive

deviations

in

expectations

make

the

responses

bigger

Adapted

from

ref

tegmental

area

-LRB-

VTA

-RRB-

and

hypothalamus

-LRB-

Hyp

-RRB-

All

these

regions

have

topographically

organized

reciprocal

connections

with

the

VTA

one

of

the

primary

dopaminergic

nuclei

in

the

brainstem

Particularly

strong

reward

responses

have

been

observed

in

the

ventral

striatum

where

numerous

studies

have

shown

that

even

abstract

proxies

for

reward

-LRB-

money

-RRB-

cause

activations

that

scale

in

proportion

to

reward

amount

or

deviation

from

an

expected

payoff

Similar

results

have

been

found

by

a

variety

of

groups

using

both

passive

and

active

games

with

monetary

A

prominent

activation

response

to

monetary

payoff

was

observed

by

Knutson

and

colleagues

in

the

NAc

and

is

shown

in

Fig.

The

NAc

like

the

OFC

and

other

parts

of

the

prefrontal

cortex

-LRB-

PFC

-RRB-

is

densely

innervated

by

dopaminergic

fibres

originating

from

neurons

housed

in

the

midbrain

Other

work

has

shown

that

simply

changing

the

predictability

of

a

stimulus

will

activate

the

NAc

and

surrounding

structures

in

the

ventral

parts

of

the

The

picture

emerging

from

this

work

is

that

responses

in

this

region

may

reflect

an

encoding

of

rewards

along

a

common

valuation

Human

critic

responses

Signal

change

-LRB-

-RRB-

with

the

states

of

the

organism

The

actor

uses

this

signal

or

a

closely

related

one

to

learn

stimulus

action

associations

so

that

actions

associated

with

higher

rewards

are

more

likely

to

be

chosen

Together

these

two

components

capture

many

features

of

the

way

that

animals

learn

basic

contingencies

between

their

actions

and

the

rewards

associated

with

those

actions

The

original

hypothesis

concerning

the

role

of

dopamine

in

reinforcement

learning

proposed

just

such

a

dual

use

of

the

reward-prediction-error

McClure

and

colleagues

recently

extended

this

original

learning

hypothesis

to

address

the

Berridge

and

Robinson

This

work

suggests

a

formal

relationship

between

the

incentive-salience

ideas

of

Berridge

and

Robinson

and

actor

critic

models

in

which

incentive

salience

is

equivalent

to

the

idea

of

expected

future

value

formalized

in

reinforcement

learning

models

-LRB-

Fig.

-RRB-

Actor

critic

models

are

now

being

used

to

address

detailed

issues

concerning

stimulus

action

For

example

extensions

to

actor

critic

models

have

addressed

the

difference

between

learning

goal-directed

approach

behaviour

and

learning

automatic

actions

-LRB-

habits

-RRB-

such

as

licking

There

are

several

behavioural

settings

that

support

the

contention

that

habit

learning

is

handled

by

different

neural

systems

from

those

involved

in

goal-directed

Dayan

and

Balleine

have

recently

offered

a

computational

extension

to

actor

critic

models

to

take

account

of

this

One

of

the

most

important

contributions

of

reinforcement

learning

theory

has

been

to

distinguish

between

the

signalling

of

the

reward

itself

and

the

computation

of

the

reward-prediction

error

Using

passive

tasks

with

a

juice

reward

reward-prediction

errors

have

been

shown

to

activate

structures

in

the

ventral

Recently

two

independent

groups

used

passive

learning

paradigms

to

visualize

reward-prediction-error

signals

in

overlapping

regions

of

the

ventral

-LRB-

Fig.

-RRB-

The

cingulate

cortex

is

another

area

that

has

been

associated

with

reinforcement

learning

signals

that

seem

to

be

reward-prediction

errors

The

error-related

negativity

-LRB-

ERN

-RRB-

is

a

scalp-recorded

event-related

potential

-LRB-

ERP

-RRB-

believed

to

originate

from

the

anterior

cingulate

cortex

that

is

consistently

observed

about

msec

following

the

commission

of

an

Similar

potentials

have

been

observed

following

negative

feedback

or

unexpected

losses

in

gambling

Holroyd

and

Coles

have

proposed

that

these

potentials

reflect

a

negative

reward-prediction-error

signal

and

this

idea

has

been

tested

under

a

variety

of

Recently

fMRI

evidence

has

suggested

that

a

region

of

anterior

cingulate

cortex

responds

under

many

of

the

same

conditions

as

the

ERN

activity

is

affected

by

both

errors

and

negative

Human

actor

responses

Rewards

critics

and

actors

in

the

human

brain

Recent

functional

magnetic

resonance

imaging

-LRB-

fMRI

-RRB-

experiments

have

used

reward

expectancy

and

conditioning

tasks

to

identify

brain

responses

that

correlate

directly

with

rewards

reward-predictionerror

signals

-LRB-

critic

-RRB-

and

signals

related

to

reward-dependent

actions

-LRB-

actor

-RRB-

Many

of

these

experiments

have

used

reinforcement

learning

models

as

a

way

to

understand

the

resulting

brain

responses

to

choose

design

details

of

the

experiment

or

to

locate

brain

responses

associated

with

specific

model

Human

reward

responses

One

implication

of

reinforcement

theory

for

behaviour

concerns

the

relationship

between

reward-prediction

errors

-LRB-

critic

signals

-RRB-

and

action

selection

-LRB-

actor

signals

-RRB-

As

discussed

in

Box

the

critic

signal

can

be

used

for

reward

learning

and

to

adjust

the

future

selection

of

reward-yielding

actions

Success

in

the

use

of

fMRI

to

detect

rewardprediction-error

signals

inspired

O'Doherty

and

colleagues

to

carry

out

a

clever

but

simple

experiment

designed

to

relate

critic

signals

to

action

The

experiment

used

a

conditioning

paradigm

that

was

carried

out

in

two

modes

The

first

required

an

action

to

obtain

a

juice

reward

and

the

second

did

not

This

experiment

showed

that

activity

in

the

dorsal

striatum

correlated

with

the

predictionerror

signal

only

when

an

action

was

needed

to

acquire

the

juice

reward

-LRB-

Fig.

-RRB-

There

was

no

similar

activity

in

this

area

when

the

juice

was

passively

delivered

This

finding

is

important

because

the

dorsal

striatum

is

involved

in

the

selection

and

sequencing

of

actions

Responses

to

rewarding

stimuli

have

been

observed

consistently

from

the

same

set

of

subcortical

regions

in

human

brains

suggesting

that

neurons

in

these

regions

respond

to

a

wide

spectrum

of

triggers

In

a

series

of

elegant

papers

Breiter

and

colleagues

used

fMRI

to

record

brain

responses

to

beautiful

visual

and

drugs

that

induce

euphoria

-LRB-

cocaine

-RRB-

The

brain

structures

they

identified

included

the

orbitofrontal

cortex

-LRB-

OFC

-RRB-

amygdala

-LRB-

Amyg

-RRB-

nucleus

accumbens

-LRB-

NAc

part

of

the

ventral

striatum

-RRB-

sublenticular

extended

amygdala

-LRB-

SLEA

part

of

the

basal

forebrain

-RRB-

ventral

NATURE

VOL

OCTOBER

Neuromodulation

and

cognitive

control

Our

consideration

of

reinforcement

learning

theory

so

far

has

focused

on

simple

situations

involving

the

association

of

a

stimulus

with

a

reward

or

with

the

selection

of

an

action

that

leads

to

an

immediate

reward

In

the

real

world

however

accrual

of

reward

may

require

an

extended

sequence

of

actions

Furthermore

we

have

considered

only

a

highly

abstracted

definition

of

the

goal

of

the

organism

the

maximization

of

cumulative

future

rewards

However

many

different

forms

of

reward

-LRB-

and

associated

actions

-RRB-

may

be

Nature

Publishing

Group

insight

review

articles

valued

by

an

organism

-LRB-

for

example

the

procurement

of

nutrition

provision

of

safety

reproduction

-RRB-

This

suggests

that

the

construct

of

a

goal

needs

to

be

refined

to

describe

the

variety

of

goal-directed

behaviour

in

which

humans

engage

The

guidance

of

behaviour

in

the

service

of

internally

represented

goals

or

intentions

is

often

referred

to

as

the

capacity

for

cognitive

control

Recent

theories

of

cognitive

control

have

elaborated

on

basic

reinforcement

learning

mechanisms

to

develop

models

that

specifically

address

the

two

challenges

suggested

above

-LRB-

-RRB-

the

need

to

learn

and

control

sequences

of

actions

required

to

achieve

a

goal

and

-LRB-

-RRB-

the

need

to

represent

the

variety

of

goals

that

an

organism

may

value

Here

we

focus

on

the

first

of

these

challenges

but

seerefs

and

for

a

discussion

of

the

latter

Prefrontal

goals

require

an

extended

sequence

of

actions

Theories

of

cognitive

control

consistently

implicate

the

PFC

as

a

site

where

representations

of

goals

are

actively

maintained

and

used

to

select

goal-directed

The

involvement

of

the

PFC

is

motivated

by

three

diverse

classes

of

observations

-LRB-

-RRB-

the

PFC

can

support

sustained

activity

in

the

face

of

distracting

-LRB-

-RRB-

damage

to

the

PFC

produces

deficits

in

goal-directed

and

-LRB-

-RRB-

the

PFC

is

selectively

engaged

by

tasks

that

rely

heavily

on

the

active

representation

of

goal

Dopamine

gating

hypothesis

Pursuit

of

a

goal

-LRB-

for

example

going

to

the

car

driving

to

the

grocery

store

or

locating

the

refrigerated

section

to

buy

milk

-RRB-

can

often

a

Right

R

b

Signal

change

-LRB-

-RRB-

Ventral

putamen

One

problem

with

the

simple

hypothesis

that

the

PFC

actively

maintains

goal

representations

is

that

this

does

not

indicate

how

or

when

this

information

should

be

updated

Failure

to

appropriately

update

goal

representations

will

lead

to

perseverative

behaviour

whereas

failure

to

adequately

maintain

them

will

result

in

distractability

Indeed

disturbances

of

the

PFC

are

known

to

be

associated

with

distractability

perseveration

or

What

is

required

is

a

mechanism

that

can

signal

when

the

goal

representation

should

be

updated

Recently

it

has

been

proposed

that

dopaminergic

signals

from

the

VTA

implement

this

mechanism

by

controlling

the

gating

of

afferent

information

into

the

PFC

-LRB-

Fig.

-RRB-

According

to

this

gating

hypothesis

the

PFC

is

resistant

to

the

influence

of

afferent

signals

in

the

absence

of

phasic

dopamine

release

allowing

it

to

preserve

the

currently

maintained

goal

representation

against

impinging

sources

of

interference

However

stimuli

that

signal

the

need

to

update

the

goal

representation

elicit

a

phasic

dopamine

response

that

opens

the

gate

and

allows

afferent

signals

to

establish

a

new

goal

representation

in

the

PFC.

Reinforcement

learning

and

working

memory

How

does

the

dopamine

system

know

which

stimuli

should

elicit

a

gating

signal

and

which

should

not

One

plausible

answer

to

this

question

comes

directly

from

the

reinforcement

learning

theory

of

dopamine

function

A

gating

signal

is

required

to

update

the

PFC

when

a

stimulus

occurs

in

the

environment

which

indicates

that

a

more

valuable

goal

can

be

achieved

if

behaviour

is

redirected

towards

that

goal

-LRB-

for

example

a

light

signalling

that

a

reward

can

be

acquired

by

going

to

some

new

location

-RRB-

In

reinforcement

learning

terms

this

corresponds

to

a

positive

reward-prediction

error

the

value

of

the

current

state

is

better

than

expected

According

to

the

reinforcement

learning

theory

of

dopamine

function

this

is

associated

with

a

phasic

burst

in

dopamine

activity

In

other

words

reinforcement

learning

theory

predicts

that

phasic

dopamine

responses

will

occur

precisely

when

needed

to

produce

a

gating

signal

Furthermore

insofar

as

the

phasic

dopamine

response

acts

as

a

learning

signal

it

will

strengthen

the

association

of

the

current

predictor

for

example

the

light

with

the

goal

representation

in

the

PFC.

It

will

also

strengthen

the

tendency

of

the

light

to

elicit

a

phasic

dopamine

response

when

it

recurs

in

the

future

The

learning

here

is

analogous

to

the

simple

light-predicts-juice

experiments

described

earlier

except

that

now

light

predicts

goal

representation

in

the

PFC

which

in

turn

leads

to

the

accrual

of

reward

This

proposal

shows

how

a

prefrontal

representation

that

plays

a

causal

role

in

the

acquisition

of

some

later

reward

comes

to

be

selected

and

reinforced

Assuming

that

dopamine

generates

both

learning

and

gating

effects

the

dopamine

system

provides

a

mechanism

for

learning

which

stimuli

should

elicit

a

gating

signal

to

update

goal

representations

in

the

PFC.

Consistent

with

this

hypothesis

the

parameter

used

to

implement

the

learning

effects

of

dopamine

in

formal

models

of

reinforcement

bears

a

remarkable

similarity

to

the

parameter

used

to

implement

gating

effects

in

models

of

dopamine-based

gating

signals

in

the

Recent

computational

modelling

work

has

demonstrated

that

implementing

concurrent

effects

of

dopamine

phasic

signals

on

reinforcement

learning

and

gating

allows

a

system

to

associate

stimuli

with

the

gating

signals

that

predict

reward

and

so

learn

how

to

update

representations

appropriately

in

the

NATURE

VOL

OCTOBER

Time

-LRB-

s

-RRB-

c

Action

required

Right

No

action

required

Figure

Detecting

actor

and

critic

signals

in

the

human

brain

using

fMRI

a

A

simple

conditioning

task

reveals

a

TD-like

prediction-error

response

-LRB-

critic

signal

-RRB-

in

the

human

brain

A

cue

is

followed

by

the

passive

delivery

of

pleasant-tasting

juice

while

subjects

are

scanned

The

highlighted

activation

is

located

in

the

ventral

part

of

the

striatum

-LRB-

the

putamen

-RRB-

a

region

known

to

respond

to

a

range

of

rewards

The

activation

represents

the

brain

response

that

correlates

with

a

continuous

TDlike

error

signal

Adapted

from

ref

b

A

similar

experimental

design

but

in

this

case

a

single

prediction

error

of

each

polarity

-LRB-

positive

and

negative

-RRB-

can

be

seen

in

the

ventral

putamen

during

a

surprising

catch

trial

Predictive

sensory

cue

-LRB-

green

arrowhead

-RRB-

normal

reward-delivery

time

-LRB-

blue

arrowhead

-RRB-

delayed

reward

time

on

catch

trials

-LRB-

red

arrowhead

-RRB-

Average

BOLD

-LRB-

blood

oxygenation

level

dependent

-RRB-

response

in

normal

trials

-LRB-

solid

line

-RRB-

and

delay

trials

-LRB-

dashed

line

-RRB-

Adapted

from

ref

c

Identification

of

actor

response

in

dorsal

striatum

A

conditioning

task

is

carried

out

in

two

modes

requiring

-LRB-

-RRB-

a

button

press

-LRB-

an

action

-RRB-

and

-LRB-

-RRB-

no

action

at

all

The

dorsal

striatum

a

region

involved

in

action

selection

responds

only

during

the

mode

where

action

is

required

and

shows

no

response

when

action

is

not

required

This

is

the

first

demonstration

of

an

actor

response

detected

in

the

human

brain

Adapted

from

ref

Nature

Publishing

Group

insight

review

articles

Prediction

error

-LRB-

t

-RRB-

guides

the

selection

of

actions

and

representations

in

target

neural

structures

Extracellular

dopamine

fluctuations

encode

-LSB-

-LRB-

t

-RRB-

-RSB-

transforms

spikes

carrying

prediction

error

into

patterns

of

neurotransmitter

release

Three-factor

update

rule

w

-LRB-

pre

-RRB-

-LRB-

post

-RRB-

-LRB-

t

-RRB-

stores

predictions

Spike-encoded

prediction

error

Dopamine

terminal

-LRB-

t

-RRB-

Midbrain

dopamine

neuron

Modified

predictions

feed

back

to

dopamine

neurons

to

influence

new

prediction

errors

Figure

The

flow

and

transformation

of

signals

carried

by

the

dopaminergic

system

This

system

is

now

thought

to

be

one

part

of

a

large

sophisticated

neural

system

for

valuation

-LRB-

-RRB-

Dopamine

neurons

encode

reward-predictionerror

signals

as

modulations

in

their

baseline

firing

rate

-LRB-

-RRB-

transformation

characterizes

the

way

in

which

modulation

of

firing

rate

changes

dopamine

delivery

-LRB-

is

known

to

be

non-linear

-RRB-

-LRB-

-RRB-

movement

of

dopamine

through

the

extracellular

space

carries

prediction-error

information

away

from

the

synapse

-LRB-

-RRB-

dopamine

delivery

to

target

structures

controls

a

range

of

functions

including

the

gating

of

working

memory

and

the

selection

of

specific

actions

-LRB-

-RRB-

any

multiplicative

learning

rule

that

depends

on

the

dopamine-encoded

prediction

error

is

able

to

store

predictions

a

vast

improvement

over

simple

storage

of

correlations

familiar

from

hebbian

learning

-LRB-

-RRB-

changes

in

target

structures

act

to

adjust

predictions

which

are

delivered

back

to

dopamine

neurons

through

long-range

connections

Recent

work

has

begun

to

explore

the

hypothesis

that

the

basal

ganglia

provide

a

mechanism

for

selective

updating

of

goal

representations

within

the

PFC.

This

proposes

that

an

important

component

of

dopaminergic

gating

takes

place

in

the

basal

ganglia

acting

selectively

on

recurrent

pathways

that

run

from

the

PFC

through

the

basal

ganglia

and

back

to

the

PFC.

Computational

models

of

the

basal

ganglia

have

shown

how

this

system

can

learn

tasks

that

require

hierarchical

updating

of

goal

representations

Neuromodulation

and

pathologies

of

cognitive

control

Reinforcement

learning

theory

provides

a

formal

framework

within

which

to

explore

quantitatively

the

effects

that

alterations

in

dopamine

function

may

have

on

behaviour

We

consider

here

two

disorders

in

which

it

has

long

been

recognized

that

dopamine

plays

a

major

role

drug

addiction

and

schizophrenia

Disturbances

of

dopamine

in

addiction

Perhaps

the

best

understood

pathology

of

dopamine

excess

is

drug

addiction

which

is

defined

as

compulsive

drug

use

despite

serious

negative

consequences

Once

a

pattern

of

compulsion

is

established

it

often

proves

remarkably

persistent

Even

when

addicted

individuals

have

been

drug-free

for

extended

periods

drug-associated

cues

can

readily

lead

to

relapse

Addictive

drugs

such

as

cocaine

amphetamine

and

heroin

all

increase

dopamine

concentrations

in

the

NAc

and

other

forebrain

structures

by

diverse

and

are

highly

reinforcing

A

new

way

to

conceptualize

the

process

of

addiction

is

in

the

terms

described

If

dopamine

plays

a

central

role

in

both

NATURE

VOL

OCTOBER

stimulus

reward

learning

and

stimulus

action

learning

and

addictive

drugs

result

in

greater

and

longer-lasting

synaptic

dopamine

concentrations

than

any

natural

reward

several

predictions

follow

Cues

that

predict

drug

availability

would

take

on

enormous

incentive

salience

by

means

of

dopamine

actions

in

the

NAc

and

PFC

and

complex

drug-seeking

behavioural

repertoires

would

be

powerfully

consolidated

by

dopamine

actions

in

the

dorsal

In

addition

dopamine

effects

in

the

PFC

may

impair

the

ability

of

the

addicted

person

to

suppress

prepotent

drug-seeking

Given

that

certain

risk-associated

behaviour

produces

phasic

dopamine

release

and

given

the

similarities

between

the

development

of

drug

addiction

and

pathologic

gambling

it

is

interesting

that

early

human

neuroimaging

results

suggest

that

similar

brain

circuits

may

be

Collectively

these

results

point

to

a

hijacking

of

dopamine

signals

in

PFC

and

limbic

structures

by

addictive

drugs

Because

these

drugs

directly

engage

dopamine-mediated

reinforcement

learning

signals

they

generate

a

feedback

loop

that

reinforces

behaviour

leading

to

drug

consumption

establishing

a

vicious

cycle

of

action

and

learning

that

explains

the

compulsive

nature

of

drug

addiction

The

degree

to

which

these

drugs

disrupt

both

phasic

and

tonic

dopamine

signals

is

not

yet

clear

However

the

reinforcement

learning

models

described

above

provide

a

framework

for

considering

possible

effects

For

the

learning

effects

over-training

with

cues

that

predict

drug

delivery

is

a

natural

consequence

of

the

role

of

phasic

dopamine

in

learning

The

PFC

gating

signal

would

also

be

unnaturally

disrupted

by

selecting

and

over-learning

grossly

maladaptive

prefrontal

representations

These

two

effects

would

conspire

to

yield

a

representation

of

the

world

that

is

grossly

biased

towards

drug-related

cues

In

addition

Nature

Publishing

Group

insight

review

articles

repeated

selection

of

maladaptive

prefrontal

representations

would

catastrophically

rearrange

the

way

in

which

normal

functions

were

categorized

within

the

PFC.

In

this

framework

the

addicted

person

's

PFC

can

no

longer

even

categorize

decision

problems

correctly

much

less

regain

control

over

the

choices

that

their

nervous

systems

deem

valuable

The

advantage

now

is

that

the

reinforcement

learning

models

provide

a

parameterized

view

of

these

problems

and

may

well

yield

new

directions

in

future

work

Disturbances

of

dopamine

in

schizophrenia

Houk

J.

C.

Adams

J.

L.

Barto

A.

G.

in

Models

of

Information

Processing

in

the

Basal

Ganglia

-LRB-

eds

Houk

J.

C.

Davis

J.

L.

Beiser

D.

G.

-RRB-

Ch

-LRB-

MIT

Cambridge

Massachusetts

-RRB-

Skinner

B.

F.

Behaviorism

at

fifty

Science

-LRB-

-RRB-

Sutton

R.

S.

Learning

to

predict

by

the

methods

of

temporal

difference

Mach

Learn

-LRB-

-RRB-

Doya

K.

Metalearning

and

neuromodulation

Neural

Netw

-LRB-

-RRB-

Dayan

P.

Abbott

L.

F.

Theoretical

Neuroscience

Ch

-LRB-

MIT

Cambridge

Massachusetts

-RRB-

Rescorla

R.

A.

Wagner

A.

R.

in

Classical

Conditioning

Current

Research

and

Theory

-LRB-

eds

Black

A.

H.

Prokasy

W.

F.

-RRB-

-LRB-

Appleton

Century-Crofts

New

York

-RRB-

Bertsekas

D.

P.

Tsitsiklis

J.

N.

in

Neuro-Dynamic

Programming

-LRB-

Athena

Scientific

Belmont

Massachusetts

-RRB-

Schultz

W.

Apicella

P.

Ljungberg

T.

Responses

of

monkey

dopamine

neurons

to

reward

and

conditioned

stimuli

during

successive

steps

of

learning

a

delayed

response

task

J.

Neurosci

-LRB-

-RRB-

Hollerman

J.

R.

Schultz

W.

Dopamine

neurons

report

an

error

in

the

temporal

prediction

of

reward

during

learning

Nature

Neurosci

-LRB-

-RRB-

Schultz

W.

Predictive

reward

signal

of

dopamine

neurons

J.

Neurophysiol

-LRB-

-RRB-

Waelti

P.

Dickinson

A.

Schultz

W.

Dopamine

responses

comply

with

basic

assumptions

of

formal

learning

theory

Nature

-LRB-

-RRB-

Bayer

H.

M.

Glimcher

P.

W.

Subjective

estimates

of

objective

rewards

using

economic

discounting

to

link

behavior

and

brain

Soc

Neurosci

Abstr

-LRB-

-RRB-

Berridge

K.

C.

Robinson

T.

E

What

is

the

role

of

dopamine

in

reward

hedonic

impact

reward

learning

or

incentive

salience

Brain

Res

Rev.

-LRB-

-RRB-

Everitt

B.

J.

et

al.

Associative

processes

in

addiction

and

reward

the

role

of

amygdala-ventral

striatal

subsystems

Ann

NY

Acad

Sci

-LRB-

-RRB-

Ikemoto

S.

Panksepp

J

The

role

of

nucleus

accumbens

dopamine

in

motivated

behavior

a

unifying

interpretation

with

special

reference

to

reward-seeking

Brain

Res

Rev.

-LRB-

-RRB-

Di

Chiara

G.

Imperato

A.

Drugs

abused

by

humans

preferentially

increase

synaptic

dopamine

concentrations

in

the

mesolimbic

system

of

freely

moving

rats

Proc

Natl

Acad

Sci

USA

-LRB-

-RRB-

Berke

J.

D.

Hyman

S.E.

Addiction

dopamine

and

the

molecular

mechanisms

of

memory

Neuron

-LRB-

-RRB-

Ikemoto

S.

Panksepp

J.

Dissociations

between

appetitive

and

consummatory

responses

by

pharmacological

manipulations

of

reward-relevant

brain

regions

Behav

Neurosci

-LRB-

-RRB-

Salamone

J.

D.

Correa

M.

Motivational

views

of

reinforcement

implications

for

understanding

the

behavioral

functions

of

nucleus

accumbens

dopamine

Behav

Brain

Res

-LRB-

-RRB-

Redgrave

P.

Prescott

T.

J.

Gurney

K.

Is

the

short-latency

dopamine

response

too

short

to

signal

reward

error

Trends

Neurosci

-LRB-

-RRB-

Egelman

D.

M.

Person

C.

Montague

P.

R

A

computational

role

for

dopamine

delivery

in

human

decision-making

J.

Cogn

Neurosci

-LRB-

-RRB-

McClure

S.

M.

Daw

N.

Montague

P.

R

A

computational

substrate

for

incentive

salience

Trends

Neurosci

-LRB-

-RRB-

Balleine

B.

W.

Dickinson

A

The

effect

of

lesions

of

the

insular

cortex

on

instrumental

conditioning

evidence

for

a

role

in

incentive

memory

Neurosci

-LRB-

-RRB-

Berridge

K.C.

in

The

Psychology

of

Learning

and

Motivation

Advances

in

Research

and

Theory

Vol

-LRB-

ed

Medin

D.

L.

-RRB-

-LRB-

Academic

San

Diego

-RRB-

Dayan

P.

Balleine

B.

W.

Reward

motivation

and

reinforcement

learning

Neuron

-LRB-

-RRB-

Berns

G.

S.

McClure

S.

M.

Pagnoni

G.

Montague

P.

R.

Predictability

modulates

human

brain

response

to

reward

J.

Neurosci

-LRB-

-RRB-

O'Doherty

J.

P.

Deichmann

R.

Critchley

H.

D.

Dolan

R.

J.

Neural

responses

during

anticipation

of

a

primary

taste

reward

Neuron

-LRB-

-RRB-

O'Doherty

J.

P.

Dayan

P.

Friston

K.

Critchley

H.

Dolan

R.

J.

Temporal

difference

models

and

reward

related

learning

in

the

human

brain

Neuron

-LRB-

-RRB-

McClure

S.

M.

Berns

G.

S.

Montague

P.

R.

Temporal

prediction

errors

in

a

passive

learning

task

activate

human

striatum

Neuron

-LRB-

-RRB-

O'Doherty

J.

P.

et

al.

Dissociable

roles

of

ventral

and

dorsal

striatum

in

instrumental

conditioning

Science

-LRB-

-RRB-

Aharon

I.

et

al.

Beautiful

faces

have

variable

reward

value

fMRI

and

behavioral

evidence

Neuron

-LRB-

-RRB-

Breiter

H.

C.

et

al.

Acute

effects

of

cocaine

on

human

brain

activity

and

emotion

Neuron

-LRB-

-RRB-

Breiter

H.

C.

Aharon

I.

Kahneman

D.

Dale

A.

Shizgal

P.

Functional

imaging

of

neural

responses

to

expectancy

and

experience

of

monetary

gains

and

losses

Neuron

-LRB-

-RRB-

Knutson

B.

Westdorp

A.

Kaiser

E.

Hommer

D.

fMRI

visualization

of

brain

activity

during

a

monetary

incentive

delay

task

Neuroimage

-LRB-

-RRB-

Knutson

B.

Adams

C.

M.

Fong

G.

W.

Hommer

D.

J.

Anticipation

of

increasing

monetary

reward

selectively

recruits

nucleus

accumbens

J.

Neurosci

-LRB-

-RRB-

Thut

G.

et

al.

Activation

of

the

human

brain

by

monetary

reward

Neuroreport

-LRB-

-RRB-

Delgado

M.

R.

Nystrom

L.

E.

Fissel

C.

Noll

D.

C.

Fiez

J.

A.Tracking

the

hemodynamic

responses

to

reward

and

punishment

in

the

striatum

J.

Neurophysiol

-LRB-

-RRB-

Elliott

R.

Friston

K.

J.

Dolan

R.

J.

Dissociable

neural

responses

in

human

reward

systems

J.

Neurosci

-LRB-

-RRB-

Montague

P.

R.

Berns

G.

S.

Neural

economics

and

the

biological

substrates

of

valuation

Neuron

-LRB-

-RRB-

Pagnoni

G.

Zink

C.

F.

Montague

P.

R.

Berns

G.

S.

Activity

in

human

ventral

striatum

locked

to

errors

of

reward

prediction

Nature

Neurosci

-LRB-

-RRB-

Gehring

W.

J.

Goss

B.

Coles

M.

G.

H.

Meyer

D.

E.

Donchin

E

A

neural

system

for

error

detection

and

compensation

Psychol

Sci

-LRB-

-RRB-

Falkenstein

M.

Hohnsbein

J.

Hoormann

J.

in

Perspectives

of

Event-Related

Potentials

Research

-LRB-

eds

Karmos

G.

et

al.

-RRB-

-LRB-

Elsevier

Science

Amsterdam

-RRB-

Gehring

W.

J.

Willoughby

A.

R

The

medial

frontal

cortex

and

the

rapid

processing

of

monetary

gains

and

losses

Science

-LRB-

-RRB-

Ullsperger

M.

von

Cramon

D.

Y.

Error

monitoring

using

external

feedback

specific

roles

of

the

habenular

complex

the

reward

system

and

the

cingulate

motor

area

revealed

by

functional

magnetic

resonance

imaging

J.

Neurosci

-LRB-

-RRB-

Disturbances

of

dopamine

function

are

also

known

to

have

a

central

role

in

schizophrenia

This

was

first

suggested

by

the

discovery

of

the

neuroleptic

drugs

that

are

effective

in

ameliorating

the

hallucinations

and

delusions

associated

with

this

illness

The

clinical

efficacy

of

these

drugs

correlates

directly

with

their

potency

in

blocking

dopaminergic

Conversely

dopamine

agonists

-LRB-

for

example

Ldopa

and

amphetamines

-RRB-

reproduce

some

of

the

same

symptoms

of

schizophrenia

Taken

together

these

results

led

to

the

hypothesis

that

schizophrenia

is

associated

with

a

hyper-dopaminergic

state

However

almost

half

a

century

of

research

has

failed

to

provide

solid

support

for

this

simple

idea

Although

neuroleptics

treat

some

of

the

more

dramatic

symptoms

of

schizophrenia

they

fail

to

treat

the

persistent

and

equally

debilitating

symptoms

of

the

disease

including

cognitive

disorganization

and

avolition

The

failure

of

the

classic

dopamine

hypothesis

is

perhaps

not

surprising

given

our

lack

of

understanding

of

the

role

that

dopamine

has

in

system-level

function

The

development

of

the

formal

models

of

dopamine

function

discussed

above

and

its

interaction

with

other

brain

systems

offers

hope

for

a

more

sophisticated

understanding

of

how

dopamine

disturbances

produce

the

patterns

of

clinical

psychopathology

observed

in

schizophrenia

For

example

along

with

evidence

of

dopamine

disturbances

it

has

long

been

recognized

that

schizophrenia

is

associated

with

disturbances

of

frontal

lobe

function

This

was

originally

suggested

by

comparing

disturbances

in

executive

function

observed

in

schizophrenia

-LRB-

for

example

distractability

and

cognitive

disorganization

-RRB-

with

those

observed

in

patients

with

frontal

lobe

damage

More

recently

neuro-imaging

studies

have

produced

more

direct

evidence

of

deficits

in

frontal

lobe

function

and

several

investigators

have

begun

to

link

these

deficits

with

disturbances

of

dopamine

function

Specifically

schizophrenia

may

be

associated

with

reduced

dopamine

activity

in

frontal

cortex

coupled

with

excess

dopamine

activity

in

subcortical

structures

such

as

the

Early

modelling

work

showed

how

a

reduction

of

dopaminergic

gain

modulation

in

the

PFC

can

simulate

the

behavioural

deficits

observed

in

patients

with

The

learning

and

gating

functions

of

dopamine

reviewed

here

suggest

ways

in

which

this

theory

could

be

elaborated

to

include

specific

neuropharmacological

findings

Perspective

Despite

our

growing

knowledge

about

some

of

the

biological

disturbances

associated

with

schizophrenia

as

yet

there

is

no

biological

assay

that

can

be

used

to

diagnose

this

disease

definitively

This

reflects

the

deep

limitations

in

our

understanding

of

the

relationship

between

biological

disturbances

and

their

clinical

expression

as

perturbed

mental

or

emotional

function

We

are

entering

a

time

where

the

formal

synthesis

of

experimental

data

both

behavioural

and

physiological

will

be

needed

to

address

the

many

open

questions

surrounding

mental

illness

and

behavioural

decision-making

doi

Sutton

R.

S.

Barto

A.

G.

Reinforcement

learning

-LRB-

MIT

Cambridge

Massachusetts

-RRB-

Montague

P.

R.

Dayan

P.

Sejnowski

T.

J

A

framework

for

mesencephalic

dopamine

systems

based

on

predictive

Hebbian

learning

J.

Neurosci

-LRB-

-RRB-

Schultz

W.

Dayan

P.

Montague

P.

R

A

neural

substrate

of

prediction

and

reward

Science

-LRB-

-RRB-

Friston

K.

J.

Tononi

G.

Reeke

G.

N.

Sporns

O.

Edelman

G.

M.

Value-dependent

selection

in

the

brain

simulation

in

a

synthetic

neural

model

Neuroscience

-LRB-

-RRB-

Nature

Publishing

Group

NATURE

VOL

OCTOBER

insight

review

articles

Nieuwenhuis

S.

Yeung

N.

Holroyd

C.

B.

Schurger

A.

Cohen

J.

D.

Sensitivity

of

electrophysiological

activity

from

medial

frontal

cortex

to

utilitarian

and

performance

feedback

Cereb

Cort

-LRB-

-RRB-

Holroyd

C.

B.

Coles

M.

G

The

neural

basis

of

human

error

processing

reinforcement

learning

dopamine

and

the

error-related

negativity

Psychol

Rev.

-LRB-

-RRB-

Holroyd

C.

B.

Nieuwenhuis

S.

Yeung

N.

Cohen

J.

D.

Errors

in

reward

prediction

are

reflected

in

the

event-related

brain

potential

Neuroreport

-LRB-

-RRB-

Holroyd

C.

B.

Larsen

J.

T.

Cohen

J.

D.

Context

dependence

of

the

event-related

brain

potential

associated

with

reward

and

punishment

Psychophysiol

-LRB-

-RRB-

Holroyd

C.

B.

et

al.

Dorsal

anterior

cingulate

cortex

shows

fMRI

response

to

internal

and

external

error

signals

Nature

Neurosci

-LRB-

-RRB-

Miller

E.

K.

Cohen

J.

D

An

integrative

theory

of

prefrontal

cortex

function

Annu

Rev.

of

Neurosci

-LRB-

-RRB-

O'Reilly

R.

C.

Braver

T.

S.

Cohen

J.

D.

in

Models

of

Working

Memory

Mechanisms

of

Active

Maintenance

and

Executive

Control

-LRB-

eds

Miyake

A.

Shah

P.

-RRB-

Ch

-LRB-

Cambridge

Univ.

Press

New

York

-RRB-

Miller

E.

K.

Li

L.

Desimone

R

A

neural

mechanism

for

working

and

recognition

memory

in

inferior

temporal

cortex

Science

-LRB-

-RRB-

Miller

E.

K.

Erickson

C.

A.

Desimone

R.

Neural

mechanisms

of

visual

working

memory

in

prefrontal

cortex

of

the

macaque

J.

Neurosci

-LRB-

-RRB-

Duncan

J.

Disorganization

of

behavior

after

frontal

lobe

damage

Cog

Neuropsychol

-LRB-

-RRB-

Shallice

T.

in

From

Neuropsychology

to

Mental

Structure

-LRB-

Cambridge

Univ.

Press

Cambridge

-RRB-

Koechlin

E.

Ody

C.

Kouneiher

F

The

architecture

of

cognitive

control

in

the

human

prefrontal

cortex

Science

-LRB-

-RRB-

Stuss

D.

T.

Knight

R.

T.

Principles

of

Frontal

Lobe

Function

-LRB-

Oxford

Univ.

Press

New

York

-RRB-

Braver

T.

S.

Cohen

J.

D.

in

Attention

and

Performance

XVIII

Control

of

Cognitive

Processes

-LRB-

eds

Monsell

S.

Driver

J.

-RRB-

-LRB-

MIT

Cambridge

Massachusetts

-RRB-

Daw

N.

D.

Kakade

S.

Dayan

P.

Opponent

interactions

between

serotonin

and

dopamine

Neural

Netw

-LRB-

-RRB-

O'Reilly

R.

C.

Noelle

D.

C.

Braver

T.

S.

Cohen

J.

D.

Prefrontal

cortex

and

dynamic

categorization

tasks

representational

organization

and

neuromodulatory

control

Cereb

Cort

-LRB-

-RRB-

Rougier

N.

P.

O'Reilly

R.

C.

Learning

representations

in

a

gated

prefrontal

cortex

model

of

dynamic

task

switching

Trends

Cogn

Sci

-LRB-

-RRB-

Wise

R.

A.

Bozarth

M.

A

A

psychomotor

stimulant

theory

of

addiction

Psychol

Rev.

-LRB-

-RRB-

Hyman

S.

E.

Malenka

R.

C.

Addiction

and

the

brain

the

neurobiology

of

compulsion

and

its

persistence

Nature

Rev.

Neurosci

-LRB-

-RRB-

Potenza

M.

N.

et

al.

Gambling

urges

in

pathological

gambling

a

functional

magnetic

resonance

imaging

study

Arch

Gen.

Psych

-LRB-

-RRB-

Cohen

B.

Dopamine

receptors

and

antipsychotic

drugs

Mclean

Hosp

J.

-LRB-

-RRB-

Weinberger

D.

R.

Implications

of

normal

brain

development

for

the

pathogenesis

of

schizophrenia

Arch

Gen.

Psych

-LRB-

-RRB-

Servan-Schreiber

D.

Printz

H.

Cohen

J.

D

A

network

model

of

catecholamine

effects

gain

signalto-noise

ratio

and

behavior

Science

-LRB-

-RRB-

Montague

P.

R.

et

al.

Dynamic

gain

control

of

dopamine

delivery

in

freely

moving

animals

J.

Neurosci

-LRB-

-RRB-

Competing

interests

statement

The

authors

declare

that

they

have

no

competing

financial

interests

NATURE

VOL

OCTOBER

Nature

Publishing

Group

reinforcement learning
neuroscience
dopamine
review