The

Journal

of

Neuroscience

November

-LRB-

-RRB-

Mini-Symposium

Do

We

Know

What

the

Early

Visual

System

Does

Matteo

Carandini

Jonathan

B.

Demb

Valerio

Mante

David

J.

Tolhurst

Yang

Dan

Bruno

A.

Olshausen

Jack

L.

Gallant

and

Nicole

C.

Smith-Kettlewell

Eye

Research

Institute

San

Francisco

California

of

Ophthalmology

and

Visual

Sciences

and

Molecular

Cellular

and

Developmental

Biology

University

of

Michigan

Ann

Arbor

Michigan

of

Physiology

University

of

Cambridge

Cambridge

United

Kingdom

Departments

of

and

Cellular

Biology

and

and

Wills

Neuroscience

Institute

and

School

of

Optometry

University

of

California

Berkeley

Berkeley

California

and

for

Neural

Science

New

York

University

New

York

New

York

We

can

claim

that

we

know

what

the

visual

system

does

once

we

can

predict

neural

responses

to

arbitrary

stimuli

including

those

seen

in

nature

In

the

early

visual

system

models

based

on

one

or

more

linear

receptive

fields

hold

promise

to

achieve

this

goal

as

long

as

the

models

include

nonlinear

mechanisms

that

control

responsiveness

based

on

stimulus

context

and

history

and

take

into

account

the

nonlinearity

of

spike

generation

These

linear

and

nonlinear

mechanisms

might

be

the

only

essential

determinants

of

the

response

or

alternatively

there

may

be

additional

fundamental

determinants

yet

to

be

identified

Research

is

progressing

with

the

goals

of

defining

a

single

standard

model

for

each

stage

of

the

visual

pathway

and

testing

the

predictive

power

of

these

models

on

the

responses

to

movies

of

natural

scenes

These

predictive

models

represent

at

a

given

stage

of

the

visual

pathway

a

compact

description

of

visual

computation

They

would

be

an

invaluable

guide

for

understanding

the

underlying

biophysical

and

anatomical

mechanisms

and

relating

neural

responses

to

visual

perception

Key

words

contrast

lateral

geniculate

nucleus

luminance

primary

visual

cortex

receptive

field

retina

visual

system

natural

images

The

ultimate

test

of

our

knowledge

of

the

visual

system

is

prediction

we

can

say

that

we

know

what

the

visual

system

does

when

we

can

predict

its

response

to

arbitrary

stimuli

How

far

are

we

from

this

end

result

Do

we

have

a

standard

model

that

can

predict

the

responses

of

at

least

some

early

part

of

the

visual

system

such

as

the

retina

the

lateral

geniculate

nucleus

-LRB-

LGN

-RRB-

or

primary

visual

cortex

-LRB-

-RRB-

Does

such

a

model

predict

responses

to

stimuli

encountered

in

the

real

world

A

standard

model

existed

in

the

early

decades

of

visual

neuroscience

until

the

it

was

given

by

the

linear

receptive

field

The

linear

receptive

field

specifies

a

set

of

weights

to

apply

to

images

to

yield

a

predicted

response

A

weighted

sum

is

a

linear

operation

so

it

is

simple

and

intuitive

Moreover

linearity

made

the

receptive

field

mathematically

tractable

allowing

the

fruitful

marriage

of

visual

neuroscience

with

image

processing

-LRB-

Robson

-RRB-

and

with

linear

systems

analysis

-LRB-

De

Valois

and

De

Valois

-RRB-

It

also

provided

a

promising

parallel

with

research

in

visual

perception

-LRB-

Graham

-RRB-

Because

it

served

as

a

standard

model

the

receptive

field

could

be

used

to

decide

which

findings

were

surprising

and

which

were

not

if

a

phenomenon

was

not

predictable

from

the

linear

receptive

field

it

was

particularly

worthy

of

publication

Research

aimed

at

testing

the

linear

receptive

field

led

to

the

discovery

of

important

nonlinear

phenomena

which

can

not

be

explained

by

a

linear

receptive

field

alone

These

phenomena

Received

Sept.

revised

Oct.

accepted

Oct.

Correspondence

should

be

addressed

to

Dr.

Matteo

Carandini

Smith-Kettlewell

Eye

Research

Institute

Fillmore

Street

San

Francisco

CA

E-mail

matteo@ski.org

DOI

Copyright

Society

for

Neuroscience

have

been

discovered

at

all

stages

of

the

early

visual

system

including

the

retina

-LRB-

for

review

see

Shapley

and

Enroth-Cugell

Demb

-RRB-

the

LGN

-LRB-

for

review

see

Carandini

-RRB-

and

area

-LRB-

for

review

see

Carandini

et

al.

Fitzpatrick

Albright

and

Stoner

-RRB-

They

have

forced

a

revision

of

the

models

based

on

the

linear

receptive

field

In

some

cases

the

revised

models

have

achieved

near

standard

model

status

for

example

the

model

of

Shapley

and

Victor

for

contrast

gain

control

in

retinal

ganglion

cells

-LRB-

Shapley

and

Victor

Victor

-RRB-

and

Heeger

's

normalization

model

of

responses

-LRB-

Heeger

-RRB-

By

and

large

however

the

discovery

of

failures

of

the

linear

receptive

field

has

deprived

the

field

of

a

simple

standard

model

for

each

visual

stage

This

review

aims

to

help

move

the

field

toward

the

definition

of

new

standard

models

bringing

the

practice

of

visual

neuroscience

closer

to

that

of

established

quantitative

fields

such

as

Physics

In

these

fields

there

is

wide

agreement

as

to

what

constitutes

a

standard

theory

and

which

results

should

be

the

source

of

surprise

The

review

is

authored

by

the

speakers

and

organizers

of

a

mini-symposium

at

the

Annual

Meeting

of

the

Society

for

Neuroscience

We

are

all

involved

in

a

similar

effort

we

construct

models

of

neurons

and

test

how

accurately

they

predict

the

responses

to

both

simple

laboratory

stimuli

and

complex

stimuli

such

as

those

that

would

be

encountered

in

nature

How

accurate

are

the

existing

models

when

held

to

a

rigorous

test

By

what

standards

should

we

judge

them

Do

they

generalize

to

large

classes

of

stimuli

How

should

the

models

be

revised

The

review

is

organized

along

the

lines

of

the

minisymposium

with

each

speaker

addressing

the

question

Do

we

understand

visual

processing

at

one

or

more

stages

of

the

visual

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

hierarchy

We

begin

with

Background

in

which

we

summarize

some

notions

that

are

at

the

basis

of

most

functional

models

in

early

vision

Demb

follows

with

an

evaluation

of

standard

models

of

the

retina

-LRB-

see

below

Understanding

the

retinal

output

-RRB-

Mante

formalizes

an

extension

to

the

linear

model

of

LGN

neurons

to

account

for

luminance

and

gain

control

adaptation

effects

-LRB-

see

below

Understanding

LGN

responses

-RRB-

The

successes

and

failures

of

cortical

models

are

addressed

by

Tolhurst

-LRB-

see

below

Understanding

simple

cells

-RRB-

and

Dan

-LRB-

see

below

Understanding

complex

cells

-RRB-

Gallant

discusses

novel

model

characterization

techniques

and

their

degree

of

success

in

areas

and

-LRB-

see

below

Evaluating

what

we

know

about

and

beyond

-RRB-

Finally

Olshausen

argues

that

our

understanding

of

is

far

from

complete

and

proposes

future

avenues

for

research

-LRB-

see

below

What

we

do

n't

know

about

-RRB-

In

Conclusion

we

isolate

some

of

the

common

ideas

and

different

viewpoints

that

have

emerged

from

these

contributions

Background

At

the

basis

of

most

current

models

of

neurons

in

the

early

visual

system

is

the

concept

of

linear

receptive

field

The

receptive

field

is

commonly

used

to

describe

the

properties

of

an

image

that

modulates

the

responses

of

a

visual

neuron

More

formally

the

concept

of

a

receptive

field

is

captured

in

a

model

that

includes

a

linear

filter

as

its

first

stage

Filtering

involves

multiplying

the

intensities

at

each

local

region

of

an

image

-LRB-

the

value

of

each

pixel

-RRB-

by

the

values

of

a

filter

and

summing

the

weighted

image

intensities

A

linear

filter

describes

the

stimulus

selectivity

for

a

neuron

images

that

resemble

the

filter

produce

large

responses

whereas

images

that

have

only

a

small

resemblance

with

the

filter

produce

negligible

responses

For

example

tuning

for

the

spatial

frequency

of

a

drifting

grating

is

described

by

the

center

surround

organization

of

filters

in

the

retina

and

LGN

-LRB-

Fig.

A

-RRB-

-LRB-

Enroth-Cugell

and

Robson

-RRB-

whereas

orientation

tuning

in

is

described

by

filters

that

are

elongated

along

one

spatial

axis

-LRB-

Fig.

B

-RRB-

-LRB-

Hubel

and

Wiesel

-RRB-

Basic

models

of

neurons

at

the

earliest

stages

of

visual

processing

-LRB-

retina

LGN

and

simple

cells

-RRB-

typically

include

a

single

linear

filter

-LRB-

Enroth-Cugell

and

Robson

Movshon

et

al.

-RRB-

whereas

models

of

neurons

at

later

stages

of

processing

-LRB-

complex

cells

and

beyond

-RRB-

require

multiple

filters

-LRB-

Fig.

-RRB-

-LRB-

Movshon

et

al.

Adelson

and

Bergen

Touryan

et

al.

-RRB-

The

second

stage

of

these

models

describes

how

the

filter

outputs

are

transformed

into

a

firing

rate

response

This

transformation

typically

takes

the

form

of

a

static

nonlinearity

-LRB-

e.g.

halfwave

rectification

-RRB-

a

function

that

depends

only

on

its

instantaneous

input

In

addition

many

models

implicitly

assume

that

firing

rate

is

expressed

into

spike

trains

via

a

Poisson

process

Although

the

receptive

field

has

been

described

thus

far

as

a

set

of

weights

arranged

in

space

-LRB-

Fig.

-RRB-

in

reality

the

concept

of

receptive

field

involves

three

dimensions

two

dimensions

of

space

and

the

dimension

of

time

The

full

spatiotemporal

receptive

field

of

a

neuron

specifies

what

weight

is

given

to

each

location

in

space

at

each

instant

in

the

recent

past

When

only

the

temporal

evolution

of

the

response

is

considered

for

a

given

spatial

position

-LRB-

Fig.

-RRB-

the

receptive

field

is

commonly

referred

to

as

a

temporal

weighting

function

Whether

they

are

specified

in

space

time

or

jointly

in

space

and

time

receptive

fields

are

typically

endowed

with

ON

and

OFF

subfields

-LRB-

Fig.

white

and

black

regions

-RRB-

An

ON

region

is

one

in

which

a

bright

light

evokes

a

positive

response

and

a

dark

light

evokes

a

negative

response

An

OFF

region

does

the

opposite

In

Figure

Basic

models

of

neurons

involved

in

early

visual

processing

In

all

models

the

response

of

a

neuron

is

described

by

passing

an

image

through

one

or

more

linear

filters

-LRB-

by

taking

the

dot

product

or

projection

of

an

image

and

a

filter

-RRB-

The

outputs

of

the

linear

filters

are

passed

through

an

instantaneous

nonlinear

function

plotted

here

as

firing

rate

on

the

ordinate

and

filter

output

on

the

abscissa

A

Simple

model

of

a

retinal

ganglion

cell

or

of

an

LGN

relay

neuron

The

model

includes

a

linear

filter

-LRB-

receptive

field

-RRB-

with

a

center

surround

organization

and

a

half-wave

rectifying

nonlinearity

Images

that

resemble

the

filter

produce

large

firing

rate

responses

whereas

images

that

resemble

the

inverse

of

the

filter

or

have

no

similarity

with

the

filter

produce

no

response

B

Model

of

a

simple

cell

as

a

filter

elongated

along

one

axis

and

a

half-wave

squaring

nonlinearity

As

in

A

only

images

that

resemble

the

filter

produce

high

firing

rate

responses

C

The

energy

model

of

a

complex

cell

The

model

includes

two

phaseshifted

linear

filters

whose

outputs

are

squared

before

they

are

summed

In

this

model

both

images

that

resemble

the

filters

and

their

inverses

produce

high

firing

rates

the

early

days

these

regions

were

called

excitatory

and

inhibitory

-LRB-

Hubel

and

Wiesel

-RRB-

However

this

name

is

misleading

their

sign

has

to

do

with

the

relative

contrast

of

light

not

to

the

operation

of

synaptic

excitation

and

inhibition

For

instance

an

OFF

region

will

deliver

substantial

excitation

in

response

to

a

dark

stimulus

-LRB-

Hirsch

-RRB-

The

advantage

of

assuming

an

initial

linear

processing

stage

is

that

it

enables

the

experimenter

to

recover

a

full

model

of

a

neuron

within

the

time

constraints

of

an

experiment

Recovering

the

filter

weights

involves

presenting

a

sufficiently

rich

stimulus

set

to

the

cell

-LRB-

e.g.

white

noise

flashed

gratings

or

natural

images

-RRB-

and

correlating

the

response

of

the

neuron

with

the

pixel

intensities

in

the

images

that

immediately

preceded

spikes

For

neurons

early

in

the

visual

system

a

single

linear

filter

is

often

extracted

by

presenting

a

random

noise

stimulus

and

computing

the

mean

pixel

intensity

before

each

spike

the

spike-triggered

average

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

Figure

The

LNP

model

of

the

spike

response

in

a

retinal

ganglion

cell

A

model

of

an

ON

Y-type

ganglion

cell

was

generated

from

s

of

response

to

a

white-noise

stimulus

contrast

was

extracellular

recording

in

in

vitro

guinea

pig

retina

methods

as

described

by

Zaghloul

et

al.

-LRB-

-RRB-

The

model

generates

a

linear

filter

-LRB-

weighting

function

-RRB-

and

a

static

nonlinearity

-LRB-

Chichilnisky

-RRB-

To

predict

the

response

to

a

novel

dataset

the

stimulus

is

passed

through

the

filter

to

generate

the

linear

model

of

the

response

The

filter

is

shown

at

a

time

when

it

closely

matched

the

stimulus

-LRB-

gray

box

-RRB-

and

so

the

linear

model

response

is

large

-LRB-

in

linear

model

units

gray

circle

-RRB-

The

linear

model

is

translated

to

a

spike

rate

using

a

static

nonlinearity

that

works

like

a

lookup

table

-LRB-

shown

in

box

-RRB-

The

point

in

the

linear

model

at

is

translated

to

a

spike

rate

of

spikes/s

The

bottom

trace

shows

the

spike

rate

-LRB-

black

line

-RRB-

to

s

of

the

novel

stimulus

-LRB-

of

s

total

-RRB-

The

test

stimulus

was

repeated

times

and

the

data

were

averaged

and

binned

-LRB-

bin

ms

-RRB-

The

gray

line

shows

the

output

of

the

LNP

model

-LRB-

bin

ms

-RRB-

The

gray

circle

shows

the

LNP

model

value

at

spikes/s

at

the

same

time

shown

above

for

the

linear

model

The

r

between

the

data

and

model

was

-LRB-

Chichilnisky

-RRB-

Similar

approaches

are

followed

in

the

sections

below

on

the

retina

lateral

geniculate

nucleus

and

simple

cells

At

later

stages

of

visual

processing

the

responses

of

multiple

linear

filters

can

be

accounted

for

by

looking

at

the

higher-order

correlations

between

random

stimuli

and

the

response

of

a

neuron

-LRB-

Simoncelli

et

al.

-RRB-

This

is

the

approach

followed

below

in

Understanding

complex

cells

Novel

nonlinear

mapping

techniques

provide

a

bridge

between

these

approaches

-LRB-

see

below

Evaluating

what

we

know

about

and

beyond

-RRB-

Understanding

the

retinal

output

The

retina

contains

a

complex

network

of

cells

divided

into

an

estimated

cell

types

photoreceptors

interneurons

and

ganglion

cells

whose

spike

trains

transmit

visual

information

to

the

rest

of

the

brain

-LRB-

Masland

Sterling

and

Demb

Wassle

-RRB-

No

predictive

model

will

suffice

for

all

types

of

ganglion

cell

because

some

cells

have

conventional

center

surround

receptive

fields

-LRB-

Fig.

A

-RRB-

-LRB-

Kuffler

Enroth-Cugell

and

Robson

-RRB-

whereas

others

have

specialized

properties

including

direction

selectivity

and

intrinsic

photosensitivity

-LRB-

Berson

Taylor

and

Vaney

Dacey

et

al.

-RRB-

As

a

starting

point

predictive

models

have

focused

on

four

ganglion

cell

types

the

ON

and

OFF-center

versions

of

sustained

-LRB-

X/parvocellular

type

-RRB-

and

transient

-LRB-

Y

magnocellular

type

-RRB-

cells

These

four

cell

types

express

relatively

simple

receptive

fields

they

project

via

the

LGN

to

visual

cortex

and

they

can

with

some

caveats

be

modeled

in

a

relatively

straightforward

way

For

the

purpose

of

the

predictive

models

in

question

we

could

ignore

all

of

the

complexity

of

retinal

circuitry

the

goal

is

simply

to

achieve

a

thorough

understanding

of

how

light

at

the

cornea

corresponds

to

spiking

responses

in

the

ganglion

cell

Perhaps

surprisingly

most

retinal

studies

have

not

attempted

to

go

all

the

way

and

predict

responses

to

natural

movies

but

rather

they

have

focused

on

a

simple

dynamic

laboratory

stimulus

white

noise

A

white-noise

stimulus

is

created

by

drawing

intensity

values

from

a

Gaussian

distribution

defined

by

a

mean

and

an

SD

of

intensity

every

ms

-LRB-

Fig.

-RRB-

White

noise

contains

approximately

equal

energy

over

a

range

of

temporal

frequencies

-LRB-

Zaghloul

et

al.

-RRB-

The

relatively

flat

temporal

frequency

spectrum

is

a

nice

feature

for

characterizing

the

receptive

field

but

this

flat

spectrum

differs

markedly

from

natural

scenes

in

which

there

is

decreasing

stimulus

energy

at

higher

temporal

frequencies

-LRB-

Simoncelli

and

Olshausen

-RRB-

Nevertheless

the

response

to

white

noise

presents

a

serious

challenge

for

predictive

models

and

reveals

several

important

nonlinearities

To

take

an

example

we

could

perform

a

simple

experiment

in

which

we

stimulate

a

cell

with

a

spot

of

light

over

the

receptive

field

center

and

modulate

the

spot

intensity

with

white

noise

-LRB-

Zaghloul

et

al.

-RRB-

In

this

case

we

build

a

model

of

the

temporal

response

of

the

cell

only

-LSB-

although

this

approach

can

easily

be

extended

to

model

the

full

spatiotemporal-chromatic

response

-LRB-

Chichilnisky

-RRB-

-RSB-

The

first

step

is

to

build

a

linear

model

of

the

response

of

the

cell

To

do

so

we

cross-correlate

the

spike

response

with

the

white-noise

stimulus

-LRB-

Sakai

and

Naka

Chichilnisky

-RRB-

The

result

is

a

linear

filter

that

represents

the

weighting

function

of

the

cell

-LRB-

see

above

Background

-RRB-

Then

at

any

instant

in

time

we

can

generate

the

linear

response

by

multiplying

the

stimulus

by

the

temporal

weighting

function

pointwise

and

summing

the

result

-LRB-

Fig.

-RRB-

To

generate

the

linear

response

at

the

next

moment

we

advance

the

temporal

weighting

function

in

time

and

repeat

the

process

Under

certain

conditions

the

linear

model

alone

predicts

the

cone

photoreceptor

response

to

a

white-noise

stimulus

-LRB-

Rieke

Baccus

and

Meister

-RRB-

However

the

linear

model

fails

for

ganglion

cell

responses

because

of

several

nonlinearities

-LRB-

Shapley

and

Victor

Victor

Chichilnisky

Kim

and

Rieke

Baccus

and

Meister

Zaghloul

et

al.

-RRB-

One

major

nonlinearity

is

the

spike

threshold

Resting

discharge

of

ganglion

cells

can

be

as

low

as

spikes/s

or

as

high

as

spikes/s

but

a

value

of

spikes/s

is

common

-LRB-

Kuffler

et

al.

Troy

and

Robson

Passaglia

et

al.

-RRB-

A

non

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

optimal

stimulus

will

reduce

the

firing

rate

but

firing

rates

can

not

go

negative

and

so

there

is

a

point

at

which

spiking

responses

are

clipped

Furthermore

an

optimal

stimulus

will

increase

the

spike

rate

but

spike

rates

can

not

be

infinitely

high

In

a

ms

period

a

cell

could

fire

at

most

approximately

four

spikes

-LRB-

or

spikes/s

-RRB-

because

of

the

ms

refractory

period

after

each

spike

Thus

the

clipping

-LRB-

rectification

-RRB-

and

the

maximum

rate

-LRB-

saturation

-RRB-

represent

two

notable

nonlinearities

These

nonlinearities

can

to

some

degree

be

modeled

as

static

meaning

that

the

linear

response

can

be

passed

through

an

input

output

function

that

is

invariant

over

time

-LRB-

Fig.

-RRB-

-LRB-

Chichilnisky

-RRB-

The

combination

of

a

linear

filter

and

a

static

nonlinearity

creates

the

linear

nonlinear

-LRB-

LN

-RRB-

model

of

spiking

-LRB-

Figs.

A

-RRB-

This

model

predicts

the

spike

rate

but

not

actual

spike

times

spiking

is

modeled

as

a

Poisson

process

defined

by

a

rate

-LRB-

with

equal

mean

and

variance

-RRB-

but

spike

times

are

otherwise

random

Thus

the

model

can

most

properly

be

termed

the

linear

nonlinear

Poisson

-LRB-

LNP

-RRB-

model

of

spiking

-LRB-

Paninski

et

al.

-RRB-

Despite

its

simplicity

the

LNP

model

works

rather

well

at

predicting

spike

rates

In

practice

one

can

generate

the

linear

prediction

of

the

response

using

the

method

described

above

One

can

estimate

the

static

nonlinear

function

by

plotting

the

linear

prediction

of

the

response

versus

the

actual

response

and

fitting

a

smooth

function

-LRB-

Chichilnisky

-RRB-

One

way

to

test

the

model

is

to

build

the

linear

and

nonlinear

stages

based

on

one

dataset

and

then

test

how

well

the

model

predicts

the

response

to

a

novel

test

stimulus

-LRB-

with

the

same

contrast

and

mean

luminance

as

the

stimulus

used

to

generate

the

model

-RRB-

On

such

tests

the

LNP

model

predicts

the

new

dataset

nearly

as

well

as

does

a

maximum

likelihood

gold

standard

-LRB-

Chichilnisky

Kim

and

Rieke

Zaghloul

et

al.

-RRB-

Another

measure

is

the

amount

of

variance

captured

by

the

model

-LRB-

r

-RRB-

In

Figure

the

LNP

model

captured

of

the

variance

in

the

spike

response

A

similar

LN

model

works

equally

well

on

subthreshold

membrane

voltage

or

current

responses

-LRB-

Kim

and

Rieke

Rieke

Baccus

and

Meister

Zaghloul

et

al.

-RRB-

We

could

feel

rather

satisfied

by

the

ability

of

the

LNP

model

to

predict

the

response

to

a

novel

test

stimulus

However

model

performance

would

degrade

quickly

if

we

changed

almost

any

aspect

of

the

test

stimulus

For

example

imagine

that

we

changed

the

contrast

-LRB-

the

SD

of

the

Gaussian

distribution

of

luminance

values

-RRB-

Increasing

contrast

reduces

the

sensitivity

of

the

linear

filter

-LRB-

height

-RRB-

and

shortens

the

integration

time

-LRB-

width

-RRB-

-LRB-

Shapley

and

Victor

Smirnakis

et

al.

Benardete

and

Kaplan

Chander

and

Chichilnisky

Kim

and

Rieke

Zaghloul

et

al.

-RRB-

Thus

to

model

the

response

at

the

new

contrast

we

would

need

to

use

a

new

filter

However

in

many

cases

we

can

model

the

response

at

the

new

contrast

with

the

same

nonlinear

function

as

before

-LRB-

Chander

and

Chichilnisky

Kim

and

Rieke

Zaghloul

et

al.

-RRB-

Still

to

predict

the

response

to

multiple

contrasts

we

would

need

to

know

the

linear

filter

for

each

contrast

Even

if

we

knew

the

linear

filter

for

all

contrast

levels

we

would

have

another

problem

Each

of

our

linear

filters

was

calculated

using

a

white-noise

stimulus

with

a

contrast

level

that

remained

constant

during

the

filter

measurement

As

soon

as

we

move

to

a

natural

stimulus

we

can

expect

that

the

contrast

would

change

continuously

and

so

we

would

need

to

know

how

the

linear

filter

changed

dynamically

over

time

with

the

contrast

level

There

is

some

evidence

that

that

the

filter

changes

rapidly

after

a

change

in

contrast

in

ms

-LRB-

Victor

Baccus

and

Meister

-RRB-

however

other

measures

suggest

a

slower

change

over

seconds

-LRB-

Kim

and

Rieke

-RRB-

Furthermore

there

are

cases

in

which

switching

contrast

to

a

new

level

changes

not

only

the

filter

but

also

the

static

nonlinearity

-LRB-

Baccus

and

Meister

-RRB-

This

introduces

a

complication

for

the

LNP

model

because

even

if

the

LNP

model

is

useful

at

a

given

steady

contrast

level

we

must

consider

that

both

the

linear

and

nonlinear

stages

would

change

dynamically

as

contrast

varied

over

time

in

a

natural

movie

The

above

example

considers

the

response

to

a

luminance

modulation

in

time

a

one-dimensional

problem

but

of

course

a

natural

movie

varies

over

time

in

two

dimensions

of

space

-LRB-

plus

there

is

the

issue

of

color

-RRB-

When

we

consider

space

two

complications

arise

First

transient

-LRB-

Y-type

-RRB-

ganglion

cells

combine

subregions

of

their

receptive

field

nonlinearly

apparently

because

of

nonlinearities

at

the

output

of

presynaptic

bipolar

cells

-LRB-

Demb

et

al.

-RRB-

Furthermore

there

are

nonlinear

signals

passed

across

the

retina

from

outside

the

classical

-LRB-

center

surround

-RRB-

receptive

field

that

are

not

captured

by

the

LNP

model

-LRB-

Demb

et

al.

Roska

and

Werblin

Olveczky

et

al.

-RRB-

Some

models

have

characterized

these

nonlinear

influences

using

quantitative

approaches

-LRB-

Shapley

and

Victor

Victor

-RRB-

However

it

is

not

clear

at

present

how

well

these

models

would

predict

responses

to

natural

stimuli

Furthermore

certain

ganglion

cells

adapt

to

the

pattern

of

light

over

space

or

time

such

that

the

linear

filter

becomes

less

sensitive

to

the

most

predictable

features

of

the

stimulus

-LRB-

Hosoya

et

al.

-RRB-

For

example

this

type

of

adaptation

would

increase

sensitivity

to

horizontal

features

after

prolonged

exposure

to

vertical

features

This

pattern

adaptation

will

need

to

be

considered

in

future

predictive

models

One

direction

to

push

the

LNP

model

is

to

generate

a

more

realistic

pattern

of

spiking

than

Poisson

output

In

fact

ganglion

cells

unlike

cortical

cells

fire

spikes

much

more

reliably

than

a

Poisson

process

-LRB-

Berry

et

al.

Reich

et

al.

Kara

et

al.

Demb

et

al.

Uzzell

and

Chichilnisky

-RRB-

For

example

a

stimulus

that

evokes

on

average

a

burst

of

nine

spikes

will

show

an

SD

-LRB-

across

repeated

trials

-RRB-

of

approximately

one

spike

rather

than

the

Poisson

value

of

three

spikes

-LRB-

i.e.

variance

of

equal

to

the

mean

-RRB-

One

recent

approach

used

a

novel

method

for

fitting

a

model

that

includes

a

linear

filter

followed

by

an

integrate-and-fire

spike

generator

-LRB-

Paninski

et

al.

-RRB-

To

model

realistic

patterns

of

spiking

the

spike

generator

includes

a

recovery

function

after

each

spike

mimicking

a

refractory

period

-LRB-

Keat

et

al.

Paninski

et

al.

-RRB-

One

result

of

this

approach

is

that

the

apparent

contrast-dependent

change

in

the

linear

filter

width

as

measured

by

the

LNP

model

may

be

an

artifact

related

to

the

refractory

period

in

the

data

-LRB-

Pillow

and

Simoncelli

-RRB-

However

intracellular

studies

which

measure

the

continuous

subthreshold

potential

suggest

that

some

amount

of

the

contrast-dependent

change

in

filter

width

may

be

real

-LRB-

Kim

and

Rieke

Zaghloul

et

al.

-RRB-

Even

given

all

of

the

above

complications

it

is

surprising

that

more

retinal

studies

have

not

attempted

to

predict

the

response

to

a

natural

movie

One

group

tested

their

model

on

a

full-field

stimulus

that

was

modulated

by

a

natural

sequence

of

light

fluctuation

and

the

model

did

a

reasonable

job

-LRB-

van

Hateren

et

al.

-RRB-

The

model

was

based

on

a

linear

filter

approach

and

included

feedback

gain

controls

to

account

for

adaptation

to

the

mean

intensity

and

contrast

and

a

rectifying

nonlinearity

to

model

the

spike

threshold

In

fact

many

bursts

of

spiking

evoked

during

the

stimulus

were

captured

by

the

model

although

there

was

clearly

room

for

improvement

Furthermore

the

study

did

not

test

the

predictive

power

of

the

model

on

novel

datasets

Still

the

results

were

generally

encouraging

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

What

are

the

next

steps

for

predictive

modeling

in

the

retina

Clearly

there

are

many

questions

left

unanswered

by

the

LNP

model

A

major

question

is

how

we

can

predict

the

linear

filter

at

any

instant

in

time

given

the

previous

statistics

of

the

stimulus

A

working

hypothesis

suggests

that

the

retina

adapts

separately

to

contrast

-LRB-

contrast

adaptation

-RRB-

and

the

mean

intensity

-LRB-

light

adaptation

-RRB-

So

one

advance

would

be

to

further

understand

the

rules

by

which

the

previous

mean

intensity

and

contrast

influence

the

filter

-LRB-

see

below

Understanding

LGN

responses

-RRB-

However

this

hypothesis

suggests

that

the

mean

and

contrast

are

the

only

relevant

parameters

and

that

these

parameters

control

filter

adaptation

independently

both

of

these

assumptions

require

additional

validation

Also

this

theory

ignores

possible

adaptation

to

higher-order

stimulus

statistics

-LRB-

Hosoya

et

al.

-RRB-

Furthermore

once

we

get

past

photoreception

and

into

the

retinal

circuit

cells

no

longer

adapt

to

light

statistics

rather

they

adapt

based

on

changes

in

neurotransmitter

release

over

time

as

well

as

intrinsic

cellular

properties

Thus

it

will

be

important

to

further

understand

cellular

mechanisms

for

adaptation

Intracellular

recordings

suggested

that

contrast

adaptation

occurs

partly

at

the

level

of

synaptic

input

and

partly

at

the

level

of

ganglion

cell

spike

generation

suggesting

an

adaptive

mechanism

intrinsic

to

the

ganglion

cell

-LRB-

Kim

and

Rieke

Zaghloul

et

al.

-RRB-

Further

understanding

cellular

mechanisms

for

adaptation

could

provide

key

insights

into

the

optimal

architecture

of

a

predictive

model

In

other

words

we

should

amend

the

statement

at

the

beginning

of

this

section

about

ignoring

retinal

circuitry

knowledge

of

the

circuitry

could

indeed

guide

the

development

of

an

appropriate

predictive

model

In

summary

the

response

to

a

dynamic

laboratory

stimulus

white

noise

can

be

predicted

fairly

well

using

a

simple

linear

model

followed

by

a

static

nonlinearity

Rapid

changes

in

stimulus

statistics

as

would

occur

in

a

natural

movie

requires

additional

understanding

of

the

rules

by

which

the

linear

and

nonlinear

stages

adapt

over

time

Furthermore

there

are

advances

to

be

made

in

modeling

spike

times

as

opposed

to

rates

and

we

need

to

further

understand

the

multiple

ganglion

cell

types

beyond

the

four

types

considered

here

To

many

in

the

field

of

visual

neuroscience

predicting

responses

in

the

retina

seems

much

simpler

than

predicting

responses

in

an

extrastriate

area

such

as

and

there

is

clearly

truth

to

this

notion

Nevertheless

there

is

still

a

ways

to

go

before

we

can

predict

retinal

responses

to

an

arbitrary

stimulus

Understanding

LGN

responses

The

LGN

occupies

a

strategic

position

a

strait

through

which

most

retinal

signals

must

pass

to

reach

visual

cortex

The

strongest

retinal

input

to

the

LGN

originates

from

ganglion

cells

of

the

X/parvocellular

type

and

of

the

Y/magnocellular

type

These

two

cell

types

have

been

studied

extensively

-LRB-

see

above

Understanding

the

retinal

output

-RRB-

and

together

constitute

of

ganglion

cells

in

the

cat

and

in

primates

-LRB-

Rodieck

et

al.

Masland

Wassle

-RRB-

Additional

input

to

LGN

relay

cells

originates

from

other

geniculate

neurons

from

subcortical

structures

and

from

cortex

-LRB-

Guillery

and

Sherman

-RRB-

It

would

be

highly

desirable

to

obtain

a

complete

description

of

how

LGN

neurons

respond

to

visual

stimuli

Such

a

description

would

summarize

the

computations

performed

by

the

retinal

and

thalamic

circuitry

and

amount

to

a

full

understanding

of

the

visual

inputs

received

by

primary

visual

cortex

The

main

determinant

of

the

responses

of

LGN

neurons

is

the

linear

receptive

field

whose

broad

attributes

are

similar

to

those

of

the

afferent

retinal

ganglion

cells

The

receptive

field

is

com

posed

of

a

center

and

of

a

larger

surround

whose

responses

interact

subtractively

-LRB-

Fig.

A

-RRB-

Both

center

and

surround

have

a

biphasic

temporal

weighting

function

-LRB-

Fig.

-RRB-

i.e.

they

weigh

contributions

from

the

recent

and

less

recent

past

with

opposite

polarity

-LRB-

Cai

et

al.

Reid

et

al.

-RRB-

The

linear

receptive

field

accurately

predicts

the

basic

selectivity

of

LGN

neuron

measured

with

gratings

For

instance

the

spatial

profile

of

the

receptive

field

predicts

the

selectivity

for

spatial

frequency

-LRB-

Kaplan

et

al.

So

and

Shapley

Shapley

and

Lennie

-RRB-

whereas

the

temporal

weighting

function

predicts

the

selectivity

for

temporal

frequency

-LRB-

Saul

and

Humphrey

Kremers

et

al.

Benardete

and

Kaplan

-RRB-

The

linear

receptive

field

does

not

describe

only

responses

to

simple

laboratory

stimuli

but

also

captures

the

basic

features

in

the

responses

to

complex

video

sequences

-LRB-

Dan

et

al.

-RRB-

The

shape

of

the

temporal

weighting

function

of

LGN

neurons

depends

on

two

strong

nonlinear

adaptive

mechanisms

that

originate

in

retina

luminance

gain

control

and

contrast

gain

control

-LRB-

see

above

Understanding

the

retinal

output

-RRB-

-LRB-

Shapley

and

Enroth-Cugell

-RRB-

These

gain

control

mechanisms

affect

the

height

-LRB-

i.e.

the

gain

-RRB-

and

width

-LRB-

i.e.

the

integration

time

-RRB-

of

the

temporal

weighting

function

Luminance

gain

control

-LRB-

also

known

as

light

adaptation

-RRB-

occurs

primarily

in

retina

It

matches

the

limited

dynamic

range

of

neurons

to

the

locally

prevalent

luminance

-LRB-

light

intensity

-RRB-

Gain

and

integration

time

are

reduced

for

locations

of

the

visual

field

where

mean

luminance

is

high

and

increased

where

mean

luminance

is

low

-LRB-

Dawis

et

al.

Rodieck

-RRB-

Contrast

gain

control

begins

in

retina

-LRB-

Shapley

and

Enroth-Cugell

Victor

Baccus

and

Meister

-RRB-

and

is

strengthened

at

subsequent

stages

of

the

visual

pathway

-LRB-

Kaplan

et

al.

Sclar

et

al.

-RRB-

It

regulates

gain

and

integration

time

on

the

basis

of

the

locally

prevalent

root-mean-square

contrast

the

SD

of

the

stimulus

luminance

divided

by

the

mean

luminance

Gain

and

integration

time

are

reduced

for

locations

of

the

visual

field

in

which

contrast

is

high

and

increased

in

which

contrast

is

low

These

gain

control

mechanisms

dampen

the

impact

of

sudden

changes

in

the

mean

luminance

or

contrast

of

a

scene

such

as

those

brought

about

by

eye

movements

This

effect

is

illustrated

in

Figure

A

and

B

by

the

responses

of

an

LGN

neuron

in

an

anesthetized

paralyzed

cat

-LRB-

Mante

et

al.

-RRB-

Stimuli

were

drifting

gratings

of

optimal

spatial

frequency

Several

seconds

after

the

onset

of

a

grating

either

mean

luminance

-LRB-

at

constant

contrast

-RRB-

or

contrast

-LRB-

at

constant

mean

luminance

-RRB-

was

suddenly

increased

LGN

responses

are

barely

affected

by

the

change

in

luminance

-LRB-

Fig.

-RRB-

and

only

weakly

affected

by

the

change

in

contrast

-LRB-

Fig.

-RRB-

Indeed

consider

the

responses

predicted

at

high

luminance

from

the

linear

receptive

field

measured

at

low

luminance

-LRB-

Fig.

red

curves

-RRB-

and

the

responses

predicted

at

high

contrast

from

the

linear

receptive

field

measured

at

low

contrast

-LRB-

Fig.

red

curves

-RRB-

The

linear

predictions

are

larger

and

slower

than

the

measured

responses

indicating

that

gain

and

integration

time

are

reduced

when

luminance

or

contrast

are

increased

This

reduction

in

gain

and

integration

time

is

completed

within

a

cycle

of

the

drifting

grating

demonstrating

that

the

gain

control

mechanisms

operate

in

ms

-LRB-

Enroth-Cugell

and

Shapley

Saito

and

Fukada

Victor

Lankheet

et

al.

Yeh

et

al.

Baccus

and

Meister

Lee

et

al.

Mante

et

al.

-RRB-

This

fast

timescale

suggests

that

gain

control

mechanisms

reduce

the

impact

of

eye

movements

which

place

the

receptive

field

of

neurons

in

the

early

visual

system

over

regions

of

widely

different

mean

luminance

and

contrast

-LRB-

Mante

et

al.

-RRB-

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

Although

the

gain

control

mechanisms

are

likely

to

play

a

major

role

during

natural

vision

most

efforts

to

predict

the

responses

of

LGN

neurons

to

natural

stimuli

have

been

limited

to

assuming

a

fixed

linear

receptive

field

-LRB-

Dan

et

al.

-RRB-

and

have

omitted

the

effects

of

gain

control

There

are

several

reasons

for

this

omission

First

existing

models

of

gain

control

are

limited

in

scope

they

operate

only

on

simplified

stimuli

such

as

gratings

and

they

lack

a

definition

of

luminance

and

contrast

that

applies

to

arbitrary

stimuli

Second

with

few

exceptions

-LRB-

Troy

and

EnrothCugell

-RRB-

luminance

gain

control

and

contrast

gain

control

were

typically

only

studied

in

isolation

During

natural

vision

however

luminance

and

contrast

vary

independently

of

each

other

-LRB-

Mante

et

al.

-RRB-

Thus

a

general

model

of

gain

control

should

predict

the

shape

of

the

temporal

weighting

function

at

every

possible

combination

of

luminance

and

contrast

Nonetheless

many

of

the

components

Figure

Predicting

responses

of

LGN

neurons

to

complex

video

sequences

A

Firing

rate

responses

of

an

LGN

neuron

to

a

drifting

grating

whose

mean

luminance

is

suddenly

increased

from

to

cd/m

-LRB-

while

contrast

is

kept

constant

-RRB-

Red

dashed

needed

to

build

a

general

model

of

gain

traces

indicate

the

prediction

of

the

linear

receptive

field

fitted

to

the

response

before

the

luminance

step

and

black

solid

traces

control

have

already

been

described

indi

indicate

the

average

response

after

the

step

B

Same

for

a

stimulus

whose

contrast

suddenly

steps

from

to

-LRB-

while

mean

vidually

For

instance

studies

have

sepa

luminance

is

kept

constant

-RRB-

C

Responses

of

an

LGN

neuron

to

a

sequence

from

Walt

Disney

's

Tarzan

Red

dashed

traces

indicate

rately

modeled

the

effects

of

luminance

the

prediction

of

the

linear

receptive

field

alone

-LRB-

measured

at

optimal

luminance

and

contrast

-RRB-

Black

solid

traces

indicate

gain

control

-LRB-

Fuortes

and

Hodgkin

prediction

of

a

nonlinear

model

In

the

nonlinear

model

the

gain

and

integration

time

of

the

receptive

field

are

regulated

by

Baylor

et

al.

Brodie

et

al.

luminance

gain

control

and

contrast

gain

control

D

Same

for

responses

to

a

Cat-cam

movie

-LRB-

Kayser

et

al.

Betsch

et

al.

Shapley

and

Enroth-Cugell

Purpura

-RRB-

In

all

panels

calibration

is

ms

and

spikes/s

and

gray

histograms

are

firing

rates

obtained

by

convolving

the

spike

et

al.

-RRB-

and

of

contrast

gain

control

trains

with

a

Gaussian

window

of

width

ms

-LRB-

SD

-RRB-

A

and

B

are

modified

from

Mante

et

al.

-LRB-

-RRB-

C

and

D

are

modified

from

-LRB-

Shapley

and

Victor

Victor

Mante

-LRB-

-RRB-

Carandini

et

al.

Benardete

and

model

of

LGN

responses

that

is

general

enough

to

predict

the

Kaplan

-RRB-

on

the

temporal

weighting

function

of

neurons

in

responses

to

arbitrary

stimuli

-LRB-

Bonin

et

al.

Mante

-RRB-

the

early

visual

system

Many

models

share

the

same

simple

deThis

model

predicts

a

number

of

nonlinear

phenomena

in

the

sign

the

temporal

weighting

function

is

obtained

by

convolving

a

responses

to

simple

stimuli

of

which

would

be

explained

by

fixed

temporal

weighting

function

with

a

variable

filter

whose

the

linear

receptive

field

alone

-LRB-

-RRB-

Response

amplitude

is

indeparameters

depend

on

luminance

or

contrast

This

design

can

be

pendent

of

mean

luminance

at

low

temporal

frequencies

aleasily

extended

to

predict

the

temporal

weighting

function

at

any

though

it

is

approximately

proportional

to

mean

luminance

at

combination

of

luminance

and

contrast

Indeed

a

recent

study

of

high

frequencies

-LRB-

Shapley

and

Enroth-Cugell

Purpura

et

LGN

responses

demonstrated

that

the

effects

of

luminance

gain

al.

-RRB-

-LRB-

-RRB-

Response

amplitude

saturates

with

contrast

As

control

and

contrast

gain

control

are

independent

of

each

other

contrast

is

increased

the

gain

is

decreased

although

not

so

much

-LRB-

Mante

et

al.

-RRB-

Thus

the

temporal

weighting

function

can

as

to

make

responses

independent

of

contrast

-LSB-

contrast

saturabe

described

by

convolving

the

fixed

weighting

function

with

two

tion

-LRB-

Derrington

and

Lennie

Cheng

et

al.

-RRB-

-RSB-

-LRB-

-RRB-

variable

filters

one

that

depends

on

luminance

and

the

other

that

Responses

are

selective

for

stimulus

size

being

maximal

for

stimdepends

on

contrast

uli

of

intermediate

size

and

being

suppressed

by

larger

stimuli

These

studies

provide

a

number

of

clues

about

how

retinal

or

-LSB-

size

tuning

-LRB-

Jones

et

al.

Solomon

et

al.

Ozeki

et

LGN

neurons

compute

the

luminance

and

contrast

of

an

arbial.

-RRB-

-RSB-

For

large

stimuli

an

increase

in

size

adds

only

little

trary

stimulus

Luminance

and

contrast

are

computed

not

only

excitatory

drive

to

the

responses

although

it

strongly

reduces

very

rapidly

-LRB-

Fig.

A

B

-RRB-

but

also

very

locally

Luminance

gain

gain

by

recruiting

more

of

the

subunits

driving

contrast

gain

control

is

driven

by

the

average

light

intensity

falling

onto

a

recontrol

-LRB-

-RRB-

The

strength

of

contrast

saturation

and

size

tuning

gion

that

is

not

larger

than

the

surround

of

the

linear

receptive

depends

on

the

temporal

frequency

of

the

stimulus

Both

are

field

-LRB-

Cleland

and

Enroth-Cugell

Enroth-Cugell

and

Shapstrong

at

low

temporal

frequencies

but

absent

at

high

temporal

ley

Enroth-Cugell

et

al.

Cleland

and

Freeman

frequencies

-LRB-

Shapley

and

Victor

Sclar

Mante

et

al.

Lankheet

et

al.

-RRB-

Similarly

contrast

gain

control

is

-RRB-

-LRB-

-RRB-

The

response

to

a

test

stimulus

is

reduced

by

superpodriven

only

by

stimuli

lying

within

the

linear

receptive

field

-LRB-

Sosition

of

a

mask

stimulus

-LSB-

masking

-LRB-

Freeman

et

al.

Bolomon

et

al.

Bonin

et

al.

-RRB-

More

precisely

contrast

nin

et

al.

-RRB-

-RSB-

seems

to

be

computed

from

the

integrated

responses

of

a

pool

of

The

nonlinear

model

predicts

the

responses

to

complex

natsmall

nonlinear

subunits

coextensive

with

the

linear

receptive

ural

stimuli

better

than

the

linear

receptive

field

alone

-LRB-

Mante

field

-LRB-

Shapley

and

Victor

Enroth-Cugell

and

Jakiela

-RRB-

For

example

the

gray

histograms

in

Figure

C

and

D

Bonin

et

al.

-RRB-

These

insights

on

gain

control

can

be

used

to

build

a

nonlinear

represent

the

firing

rate

of

an

LGN

neuron

in

response

to

two

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

complex

stimuli

movies

taken

from

the

head

of

a

cat

roaming

through

a

forest

-LSB-

Cat-cam

-LRB-

Kayser

et

al.

Betsch

et

al.

-RRB-

-RSB-

and

segments

of

cartoons

-LRB-

Walt

Disney

's

Tarzan

-RRB-

The

linear

receptive

field

which

has

the

same

temporal

weighting

function

throughout

the

movies

predicts

the

basic

features

of

the

response

but

not

the

details

-LRB-

red

curves

-RRB-

In

particular

it

captures

the

timing

of

the

responses

but

not

their

amplitude

The

predictions

of

the

nonlinear

model

in

which

gain

and

integration

time

are

adjusted

dynamically

are

more

accurate

than

those

of

the

linear

receptive

field

-LRB-

black

curves

-RRB-

Because

of

luminance

gain

control

the

predictions

of

the

nonlinear

model

tend

to

be

higher

than

those

of

the

linear

model

during

the

dark

Tarzan

movie

and

lower

during

the

bright

Cat-cam

movie

Because

of

contrast

gain

control

the

two

models

make

different

predictions

about

the

relative

magnitude

of

the

responses

within

a

movie

The

comparison

between

the

simpler

linear

model

and

the

more

complex

nonlinear

model

is

fair

because

the

models

were

given

the

same

number

of

free

parameters

-LRB-

two

spontaneous

firing

rate

and

maximal

firing

rate

-RRB-

The

remaining

parameters

were

estimated

from

the

responses

to

gratings

and

then

fixed

in

the

predictions

of

the

responses

to

complex

stimuli

Fixing

the

parameters

in

advance

is

necessary

to

compare

the

predictions

on

an

equal

footing

the

more

complex

nonlinear

model

does

not

necessarily

have

to

predict

the

data

better

than

the

simpler

linear

model

In

fact

given

the

complexity

of

the

nonlinear

model

it

would

have

been

difficult

to

estimate

its

parameters

directly

from

the

responses

to

complex

stimuli

This

approach

might

be

useful

to

characterize

also

later

stages

of

visual

processing

in

which

neurons

exhibit

progressively

more

nonlinear

properties

Even

the

nonlinear

model

however

fails

to

capture

some

features

in

the

responses

In

particular

the

measured

responses

tend

to

be

more

transient

than

the

predicted

responses

One

factor

contributing

to

the

transient

responses

could

be

the

mechanisms

generating

bursts

of

actions

potentials

after

a

hyperpolarization

lasting

ms

or

longer

LGN

neurons

are

likely

to

respond

to

a

depolarization

with

a

burst

of

action

potentials

that

is

not

predicted

by

simple

rectification

of

the

membrane

potential

-LRB-

for

review

see

Sherman

-RRB-

Bursts

are

a

prominent

feature

of

LGN

responses

in

anesthetized

or

sleeping

animals

but

less

so

in

awake

animals

-LRB-

Guido

and

Weyand

Ramcharan

et

al.

-RRB-

Another

factor

contributing

to

the

transient

responses

lies

in

the

spike

generation

mechanisms

firing

rates

are

more

transient

than

predicted

by

a

simple

rectification

of

the

membrane

potential

-LRB-

see

above

Understanding

the

retinal

output

-RRB-

Both

mechanisms

could

be

easily

incorporated

into

the

nonlinear

model

-LRB-

Mukherjee

and

Kaplan

Smith

et

al.

Keat

et

al.

Lesica

and

Stanley

Paninski

et

al.

-RRB-

Finally

the

nonlinear

model

might

be

easily

extended

to

capture

also

the

nonlinear

spatial

properties

of

Y-cells

In

fact

at

least

in

retina

the

output

of

Y-cells

can

be

thought

of

as

the

sum

of

a

pool

of

nonlinear

subunits

similar

to

the

one

driving

contrast

gain

control

-LRB-

Hochstein

and

Shapley

Victor

and

Shapley

Enroth-Cugell

and

Freeman

Demb

et

al.

-RRB-

In

summary

there

is

now

a

fairly

good

understanding

of

the

linear

and

nonlinear

components

required

to

model

responses

of

the

broad

majority

of

LGN

neurons

Many

nonlinear

properties

of

LGN

neurons

can

be

captured

by

a

single

model

that

is

general

enough

to

operate

on

arbitrary

stimuli

that

vary

in

both

space

and

time

This

model

will

be

a

useful

tool

to

explore

the

effects

of

gain

control

during

natural

vision

Once

extended

with

bursting

and

spiking

mechanisms

it

promises

to

provide

a

tractable

description

of

the

responses

of

LGN

neurons

and

thus

of

the

input

to

primary

visual

cortex

Understanding

simple

cells

Simple-cell

receptive

fields

were

first

described

in

area

of

the

cat

by

Hubel

and

Wiesel

-LRB-

-RRB-

who

defined

them

as

follows

these

fields

were

termed

simple

because

like

retinal

and

geniculate

fields

-LRB-

-RRB-

they

were

subdivided

into

distinct

excitatory

and

inhibitory

regions

-LRB-

-RRB-

there

was

summation

within

the

separate

excitatory

and

inhibitory

parts

-LRB-

-RRB-

there

was

antagonism

between

excitatory

and

inhibitory

regions

and

-LRB-

-RRB-

it

was

possible

to

predict

responses

to

stationary

or

moving

spots

of

various

shapes

from

a

map

of

the

excitatory

and

inhibitory

areas

-LRB-

Hubel

and

Wiesel

-RRB-

If

a

neuron

failed

any

part

of

this

four-part

definition

-LRB-

particularly

point

-RRB-

then

it

would

be

termed

a

complex

cell

These

definitions

were

qualitative

many

subsequent

studies

have

enquired

whether

successful

quantitative

definitions

are

possible

Point

is

crucial

to

the

definition

can

a

straightforward

receptive-field

map

predict

how

the

neuron

responds

to

other

visual

stimuli

We

must

first

acknowledge

that

predicting

responses

to

time-varying

stimuli

-LRB-

e.g.

moving

ones

-RRB-

requires

knowledge

of

the

time

courses

of

responses

in

different

parts

of

the

receptive

field

As

explained

above

in

Background

a

static

receptive

field

should

be

replaced

by

a

spatiotemporal

receptive

field

map

-LRB-

McLean

and

Palmer

Reid

et

al.

DeAngelis

et

al.

-RRB-

which

documents

differences

in

response

time

course

-LRB-

impulse

or

step

response

-RRB-

in

different

parts

of

the

field

-LRB-

Movshon

et

al.

Dean

and

Tolhurst

-RRB-

The

essence

of

prediction

is

the

same

but

strictly

the

field

and

stimuli

should

be

considered

as

functions

of

time

as

well

as

functions

of

space

Those

who

follow

Hubel

and

Wiesel

's

definitions

of

simple

and

complex

cells

generally

find

the

two

classes

of

neuron

in

approximately

equal

numbers

in

The

clear

definition

of

a

simple

cell

has

been

massively

influential

in

visual

science

because

it

offers

the

promise

that

from

relatively

simple

experiments

we

may

understand

how

approximately

half

of

the

neurons

in

would

respond

in

more

complex

situations

such

as

viewing

of

natural

scenes

The

definition

says

essentially

that

spatiotemporal

summation

in

simple

cells

is

linear

only

very

simple

arithmetic

is

needed

to

calculate

how

a

given

simple

cell

will

respond

to

some

arbitrary

stimulus

Such

is

the

starting

point

for

modeling

human

psychophysical

experiments

-LRB-

Watson

-RRB-

or

for

hypothesizing

how

natural

information

may

be

most

efficiently

coded

in

-LRB-

Willmore

and

Tolhurst

-RRB-

The

simple-cell

definition

offers

so

much

that

we

are

reluctant

to

ask

whether

it

really

works

This

section

asks

whether

simple

experiments

on

simple

cells

really

do

allow

quantitative

predictions

about

the

responses

to

other

more

complicated

stimuli

Movshon

et

al.

-LRB-

b

-RRB-

examined

the

linearity

of

spatial

summation

in

simple

and

complex

cells

in

cat

followed

by

Andrews

and

Pollen

-LRB-

-RRB-

These

studies

compared

spatial

receptive-field

maps

with

the

tuning

for

sinusoidal

gratings

and

found

that

important

aspects

of

summation

in

simple

cells

were

indeed

linear

when

tested

quantitatively

Later

studies

have

convincingly

shown

that

the

spatiotemporal

receptive

field

of

a

simple

cell

precisely

predicts

the

optimal

orientation

spatial

frequency

and

temporal

frequency

of

sinusoidal

gratings

of

the

neuron

-LRB-

Movshon

et

al.

Jones

and

Palmer

Tadmor

and

Tolhurst

DeAngelis

et

al.

Gardner

et

al.

-RRB-

However

even

the

experiments

noted

nonlinearities

in

simple-cell

behavior

A

saving

device

was

to

consider

the

simple

cell

as

a

black

box

with

two

stages

a

linearly

summating

stage

followed

by

one

or

more

nonlinearities

that

do

not

affect

the

underlying

initial

linear

sum

-LRB-

Fig.

B

-RRB-

Although

the

spatiotemporal

receptive

field

of

a

cell

predicts

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

the

cell

's

optimal

stimulus

well

it

poorly

predicts

the

relative

magnitude

of

responses

to

nonoptimal

stimuli

-LRB-

e.g.

it

overestimate

the

bandwidths

of

orientation

and

frequency

tuning

curves

-RRB-

-LRB-

Jones

and

Palmer

Tadmor

and

Tolhurst

DeAngelis

et

al.

Gardner

et

al.

-RRB-

Linear

models

also

fail

to

explain

the

relative

magnitudes

of

response

in

the

two

directions

of

movement

orthogonal

to

the

preferred

orientation

of

the

neuron

-LRB-

Albrecht

and

Geisler

Reid

et

al.

-RRB-

These

failures

of

the

linear

model

have

an

easy

rationalization

in

the

two-stage

model

-LRB-

Fig.

B

-RRB-

Most

experiments

record

neuronal

response

extracellularly

as

trains

of

action

potentials

but

the

membrane

potential

changes

of

neurons

must

exceed

a

threshold

before

spiking

activity

is

evident

-LRB-

Carandini

and

Ferster

-RRB-

Many

failures

of

the

simple

linear

model

can

be

accounted

for

arithmetically

by

supposing

that

a

simple

linear

sum

is

transformed

at

the

output

of

the

neuron

by

passage

through

a

nonlinear

transducer

function

this

may

simply

have

a

threshold

nonlinearity

or

might

be

a

sigmoidal

function

of

stimulus

contrast

-LRB-

Schumer

and

Movshon

Tolhurst

and

Dean

Tadmor

and

Tolhurst

Albrecht

and

Geisler

DeAngelis

et

al.

-RRB-

Indeed

intracellular

recordings

in

simple

cells

-LRB-

which

presumably

show

the

black

box

before

any

nonlinear

output

transform

-RRB-

do

suggest

that

the

strength

of

directional

selectivity

and

the

orientation

tuning

bandwidth

can

be

described

by

a

linear

model

in

the

first

stage

of

the

black

box

-LRB-

Jagadeesh

et

al.

Lampl

et

al.

-RRB-

Post

hoc

application

of

a

nonlinear

transducer

to

the

linear

prediction

may

work

arithmetically

but

there

are

inconsistencies

between

experiments

-LRB-

Tolhurst

and

Heeger

-RRB-

Furthermore

there

are

other

nonlinear

behaviors

that

can

not

be

explained

in

such

a

way

Notable

nonlinearities

-LRB-

shared

with

complex

cells

-RRB-

are

response

saturation

at

high

contrasts

-LRB-

Albrecht

and

Hamilton

-RRB-

and

nonspecific

suppression

-LRB-

Bonds

DeAngelis

et

al.

Tolhurst

and

Heeger

-RRB-

in

which

the

response

of

a

simple

cell

to

its

optimal

stimulus

is

suppressed

by

simultaneously

presenting

stimuli

that

evoke

no

overt

response

when

presented

alone

Heeger

-LRB-

b

-RRB-

proposed

a

neuronal

circuit

that

embraces

these

and

many

other

nonlinear

behaviors

of

simple

cells

essentially

each

simple

cell

performs

a

first-stage

linear

sum

of

its

spatiotemporal

inputs

half-squares

that

linear

sum

giving

an

energy

response

-LRB-

half-squaring

achieves

much

the

same

as

a

threshold

nonlinearity

-RRB-

and

is

then

subject

to

divisive

inhibition

from

all

other

neurons

whose

receptive

fields

cover

the

same

part

of

visual

field

The

divisive

inhibition

gives

rise

to

contrast

normalization

Application

of

this

model

-LRB-

Heeger

Tolhurst

and

Heeger

-RRB-

resolves

subtle

failures

-LRB-

Reid

et

al.

Tolhurst

and

Dean

-RRB-

in

the

predictions

of

the

relative

magnitudes

of

response

to

moving

and

stationarymodulated

gratings

which

can

not

be

resolved

by

simply

running

a

linear

response

sum

through

a

nonlinear

output

transducer

Elaborations

of

the

contrast-normalization

model

-LRB-

Carandini

and

Heeger

Carandini

et

al.

-RRB-

have

embraced

additional

nonlinear

behaviors

-LRB-

previously

unaccounted

-RRB-

such

as

phase

advance

at

high

contrasts

-LRB-

Dean

and

Tolhurst

-RRB-

The

contrast-normalization

model

has

been

influential

in

psychophysical

modeling

-LRB-

Watson

and

Solomon

-RRB-

as

well

as

in

understanding

the

details

of

simple-cell

-LRB-

and

complex-cell

-RRB-

responses

it

is

ironic

that

its

proponents

now

suggest

a

very

different

neurophysiological

mechanism

-LRB-

Carandini

et

al.

Freeman

et

al.

-RRB-

although

the

arithmetic

remains

more

or

less

the

same

Nonspecific

suppression

results

from

stimuli

within

the

receptive

field

There

is

another

nonlinearity

sometimes

confused

Figure

Predicting

responses

of

simple

cells

to

complex

images

A

The

receptive

field

of

a

ferret

simple

cell

-LRB-

see

Fig.

B

-RRB-

The

dashed

lines

outline

the

field

for

comparison

in

B

and

C.

B

The

digitized

photograph

that

evoked

the

largest

OFF

response

in

this

simple

cell

C

A

Gabor

patch

that

approximately

matches

the

disposition

of

the

actual

receptive

field

in

A.

D

The

ordinate

plots

the

actual

responses

of

the

simple

cell

to

different

photographs

-LRB-

average

of

presentations

each

measured

in

ms

bins

-RRB-

Positive

values

are

ON

responses

spikes

generated

during

the

ms

presentation

Negative

values

are

OFF

responses

spikes

generated

on

removal

of

the

photograph

The

abscissa

shows

the

responses

predicted

on

a

totally

linear

model

which

includes

solely

the

receptive

field

shown

in

C

and

not

the

nonlinearity

of

the

output

-LRB-

see

Fig.

B

-RRB-

with

it

stimuli

outside

the

classical

receptive

field

of

a

simple

cell

may

also

suppress

or

facilitate

its

responses

to

its

preferred

stimuli

as

first

described

by

Blakemore

and

Tobin

-LRB-

-RRB-

and

Maffei

and

Fiorentini

-LRB-

-RRB-

It

is

growing

clearer

that

different

mechanisms

of

suppression

are

involved

within

the

classical

receptive

field

and

outside

-LRB-

Sengpiel

et

al.

Freeman

et

al.

Li

et

al.

Sengpiel

and

Vorobyov

-RRB-

but

we

do

not

yet

have

simple

arithmetic

rules

to

describe

these

nonlinearities

Suppression

or

facilitation

from

outside

the

classical

receptive

field

may

result

from

local

connections

within

or

from

feedback

from

more

anterior

visual

areas

perhaps

subserving

selective

attention

or

perceptual

grouping

There

is

a

large

literature

on

this

topic

that

is

beyond

the

present

scope

-LRB-

for

review

see

Fitzpatrick

Freeman

et

al.

Chisum

and

Fitzpatrick

-RRB-

Thus

many

nonlinearities

of

simple-cell

behavior

are

not

evident

from

the

receptive-field

structure

but

they

can

be

accommodated

neatly

into

the

two-stage

black

box

the

linearly

summing

first

stage

is

followed

by

half-squaring

and

contrast

normalization

The

arithmetic

is

fairly

easy

and

it

works

well

However

how

important

are

all

of

these

nonlinearities

in

the

overall

behavior

of

the

neurons

Smyth

et

al.

-LRB-

-RRB-

examined

how

simple

cells

in

anesthetized

paralyzed

ferret

respond

to

ms

flashes

of

digitized

photographs

of

natural

scenes

to

understand

how

neurons

might

respond

under

natural

vision

Others

have

also

sought

to

understand

how

the

receptive

field

structure

or

grating

responses

of

simple

cells

relate

to

their

responses

to

complex

natural

scene

stimuli

-LRB-

Ringach

et

al.

Vinje

and

Gallant

Weliky

et

al.

David

et

al.

-RRB-

Figure

A

shows

the

spatial

receptive

field

of

one

simple

cell

recorded

by

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

Smyth

et

al.

-LRB-

-RRB-

mapped

with

small

bright

and

dark

squares

According

to

point

of

the

simple-cell

definition

we

expect

this

neuron

to

respond

particularly

well

to

a

bright-dark

border

slightly

off

horizontal

and

toward

the

top

right

of

the

stimulus

area

Figure

B

shows

the

photograph

that

elicited

-LRB-

by

far

-RRB-

the

most

activity

from

this

neuron

There

is

indeed

just

the

border

predicted

although

its

polarity

is

reversed

compared

with

the

ON

and

OFF

regions

of

the

field

the

photograph

evoked

strong

OFF

responses

Figure

shows

a

Gabor

function

model

of

the

receptive

field

-LRB-

Field

and

Tolhurst

Jones

and

Palmer

-RRB-

fitted

by

eye

It

was

used

to

estimate

how

a

totally

linear

field

might

respond

to

the

photographs

presented

no

nonlinearities

are

modeled

in

the

calculation

Figure

D

plots

the

actual

response

of

the

simple

cell

to

the

photographs

against

the

responses

predicted

from

the

stylized

linearly

summating

field

The

linear

model

conveys

the

gist

of

the

actual

responses

-LRB-

r

-RRB-

there

are

no

astonishing

outliers

In

truth

the

simple

cell

of

Figure

is

the

one

whose

responses

to

natural

scenes

were

best

predicted

by

linear

modeling

-LRB-

Smyth

et

al.

-RRB-

However

the

results

for

this

and

other

simple

cells

suggest

that

although

output

nonlinearities

may

reduce

response

magnitudes

below

the

linear

prediction

there

is

little

evidence

here

that

nonlinear

effects

could

fundamentally

alter

the

basic

trigger

features

for

activating

a

simple

cell

It

is

important

also

to

recognize

that

simple

cells

are

heterogeneous

some

simple

cells

may

differ

little

from

complex

cells

-LRB-

Dean

and

Tolhurst

Mechler

and

Ringach

-RRB-

Movshon

et

al.

-LRB-

-RRB-

described

some

nonlinear

simple

cells

in

which

the

ON

and

OFF

receptive-field

regions

overlap

Dean

and

Tolhurst

-LRB-

-RRB-

found

that

receptive-field

structure

was

continuously

graded

from

simple

cells

exactly

fitting

the

Hubel

and

Wiesel

definition

through

nonlinear

simple

cells

and

discrete

complex

cells

to

frank

complex

cells

-LRB-

Priebe

et

al.

Mata

and

Ringach

-RRB-

Indeed

simple

and

complex

cells

may

not

form

a

dichotomy

at

all

Of

course

this

is

not

to

say

that

all

cells

are

the

same

for

instance

it

is

clearly

understood

that

cells

at

the

simple

end

of

the

continuum

are

found

in

different

cortical

layers

than

those

at

the

complex

end

-LRB-

Martinez

et

al.

-RRB-

Receptive-field

mapping

techniques

typically

subtract

the

responses

say

to

dark

stimuli

from

those

to

bright

stimuli

so

that

the

receptive

field

seems

to

be

single

valued

at

each

point

The

resulting

linear

receptive

field

is

an

incomplete

reflection

of

the

overall

responses

of

the

neuron

For

the

varied

population

of

simple

cells

it

is

unclear

what

proportion

of

response

is

dependent

only

on

the

idealized

linear

receptive

field

and

what

has

been

obscured

by

ignoring

the

inherent

nonlinearities

of

summation

The

two-stage

model

of

Figure

B

is

inaccurate

its

simplicity

and

convenience

may

be

misleading

Geniculate

inputs

are

inherently

nonlinear

and

may

be

subject

to

depression

leading

to

the

nonspecific

suppression

noted

above

-LRB-

Carandini

et

al.

-RRB-

Nonlinear

inputs

would

result

in

inherent

nonlinearities

of

simple-cell

summation

unless

say

there

were

push

pull

inhibition

-LRB-

Glezer

et

al.

-RRB-

The

role

of

such

inhibition

has

been

explored

further

-LRB-

Palmer

and

Davis

Tolhurst

and

Dean

Ferster

Tolhurst

and

Dean

Hirsch

et

al.

-RRB-

In

particular

Tolhurst

and

Dean

-LRB-

-RRB-

and

Atick

and

Redlich

-LRB-

-RRB-

proposed

that

breakdown

of

push

pull

inhibition

underlies

the

appearance

of

nonlinearities

of

spatial

summation

even

in

the

first

stage

of

the

two-stage

model

-LRB-

Fig.

-RRB-

Indeed

all

that

may

distinguish

many

complex

cells

from

simple

cells

might

just

be

the

strength

of

the

inhibitory

signals

that

mask

inherently

nonlinear

summation

-LRB-

Wielaard

et

al.

Mechler

and

Ringach

-RRB-

Figure

Toward

a

complete

model

of

simple

cells

A

popular

conception

of

simple-cell

response

behavior

involves

a

first

stage

that

shows

strictly

linear

spatiotemporal

summation

and

a

subsequent

stage

may

be

subject

to

a

variety

of

nonlinear

phenomena

which

do

not

impinge

on

the

fundamental

linearity

of

the

first

stage

-LRB-

see

Fig.

B

-RRB-

This

schematic

shows

a

revision

of

this

convenient

model

which

includes

a

number

of

nonlinear

mechanisms

Some

of

these

mechanisms

-LRB-

those

depicted

as

affecting

the

output

nonlinearity

-RRB-

spare

the

fundamental

linearity

of

summation

The

remaining

ones

however

cause

nonlinear

summation

which

differs

only

in

degree

from

the

obvious

nonlinear

behavior

of

complex

cells

Literal

adherence

to

point

of

the

definition

of

a

simple

cell

means

that

any

significant

failure

of

prediction

would

require

the

neuron

to

be

reclassified

as

a

complex

cell

Thus

we

would

be

bound

to

understand

simple

cells

any

problematic

neuron

must

be

a

complex

cell

Failure

of

the

linear

receptive

field

model

is

not

a

problem

for

neurophysiologists

but

it

is

for

those

computational

modelers

who

would

like

all

neurons

in

the

visual

cortex

to

be

described

in

a

few

lines

of

elegant

code

with

their

receptivefield

parameters

neatly

spaced

along

a

theorist

's

dimensions

We

need

not

confuse

failure

of

the

attractive

linear

receptive

field

model

of

the

simple

cell

with

failure

to

understand

the

visual

cortex

as

a

whole

-LRB-

compare

with

below

What

we

do

n't

know

about

-RRB-

Many

of

the

bolt-on

nonlinearities

of

simple

cell

behavior

can

be

parameterized

coherently

the

failures

of

push

pull

are

an

irritation

to

modelers

but

there

is

no

need

to

believe

that

they

portend

any

dramatic

change

in

neuronal

behavior

over

the

linear

model

and

most

significantly

much

progress

has

been

made

in

understanding

how

-LRB-

frankly

nonlinear

-RRB-

complex

cells

respond

to

naturalistic

stimuli

-LRB-

compare

with

below

Understanding

complex

cells

-RRB-

Understanding

complex

cells

Although

the

orientation

and

spatial

frequency

selectivity

of

each

simple

cell

is

directly

related

to

the

spatial

profile

of

its

receptive

field

which

consists

of

elongated

ON

and

OFF

subregions

-LRB-

see

above

Understanding

simple

cells

-RRB-

for

complex

cells

this

relationship

is

far

less

obvious

A

complex

cell

usually

exhibits

mixed

ON

and

OFF

responses

throughout

its

receptive

field

-LRB-

Hubel

and

Wiesel

-RRB-

For

example

the

response

of

the

cell

to

a

bar

stimulus

depends

on

both

the

orientation

and

width

of

the

bar

in

a

manner

similar

to

simple

cells

but

the

cell

responds

indiscriminately

to

light

and

dark

bars

as

long

as

the

bar

stands

out

from

the

gray

background

Such

insensitivity

to

contrast

polarity

is

an

important

form

of

nonlinearity

that

renders

spike-triggered

average

ineffective

for

measuring

complex-cell

receptive

fields

Significant

progress

in

understanding

complex-cell

receptive

fields

was

first

made

by

measuring

the

nonlinear

interactions

between

a

pair

of

bars

at

the

preferred

orientation

of

the

cell

-LRB-

Movshon

et

al.

Emerson

et

al.

-RRB-

These

studies

have

revealed

the

existence

of

subunits

of

the

complex-cell

receptive

field

whose

spatial

structure

can

predict

the

frequency

tuning

of

the

cell

More

recently

an

alternative

method

has

been

used

to

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

characterize

complex-cell

receptive

fields

that

uses

large

ensembles

-LRB-

tens

of

thousands

-RRB-

of

complex

visual

stimuli

such

as

white

noise

or

natural

images

and

a

spike-triggered

covariance

-LRB-

STC

-RRB-

analysis

to

receptive-field

estimation

As

a

first

step

in

STC

the

stimulus

preceding

each

recorded

spike

is

collected

to

form

the

spike-triggered

stimulus

ensemble

just

like

in

spike-triggered

average

Then

the

covariance

matrix

instead

of

the

mean

of

this

spike-triggered

ensemble

is

computed

Eigenvectors

of

this

matrix

with

significant

eigenvalues

-LRB-

those

significantly

different

from

the

control

eigenvalues

calculated

based

on

random

spike

trains

-RRB-

represent

visual

features

that

directly

affect

the

neuronal

response

This

method

has

been

used

effectively

to

analyze

the

nonlinear

response

properties

of

fly

visual

neurons

-LRB-

De

Ruyter

Van

Steveninck

and

Bialek

Brenner

et

al.

-RRB-

and

the

receptive

fields

of

mammalian

complex

cells

with

either

random-bar

stimuli

aligned

to

the

preferred

orientation

of

the

cell

-LRB-

Touryan

et

al.

Rust

et

al.

-RRB-

or

natural

images

presented

in

a

random

temporal

sequence

-LRB-

Fig.

A

-RRB-

-LRB-

Touryan

et

al.

-RRB-

When

used

with

natural

images

this

method

needs

to

be

modified

to

correct

for

the

spatial

correlations

in

the

images

-LRB-

Field

Dong

and

Atick

Simoncelli

and

Olshausen

-RRB-

In

addition

to

STC

complex-cell

receptive

fields

have

been

analyzed

with

a

phase-separated

Fourier

model

-LRB-

see

below

Evaluating

what

we

know

about

and

beyond

-RRB-

supervised

training

of

neural

networks

-LRB-

Lehky

et

al.

Lau

et

al.

Prenger

et

al.

-RRB-

least-square

algorithms

-LRB-

Ringach

et

al.

-RRB-

and

information

maximization

-LRB-

Sharpee

et

al.

-RRB-

Here

we

will

focus

the

discussion

on

the

results

of

the

STC

analysis

For

most

of

the

complex

cells

in

cat

STC

identified

two

significant

eigenvectors

-LRB-

Touryan

et

al.

-RRB-

each

of

which

corresponds

to

a

stimulus

pattern

-LRB-

referred

to

as

a

feature

-RRB-

that

is

effective

for

driving

the

cell

Each

feature

exhibits

ON

and

OFF

subregions

resembling

the

receptive

fields

of

simple

cells

-LRB-

Hubel

and

Wiesel

Jones

and

Palmer

DeAngelis

et

al.

-RRB-

The

spatial

profiles

of

the

significant

eigenvectors

along

the

axis

perpendicular

to

the

preferred

orientation

of

the

cell

can

be

well

approximated

by

Gabor

functions

and

the

two

significant

eigenvectors

of

each

cell

exhibit

similar

spatial

frequencies

but

a

phase

difference

of

-LRB-

Fig.

B

-RRB-

After

identifying

these

visual

features

for

each

cell

the

next

step

is

to

quantify

the

contribution

of

each

feature

to

the

response

of

the

cell

This

is

achieved

by

computing

the

contrast

response

function

of

each

significant

eigenvector

-LRB-

Chichilnisky

Touryan

et

al.

-RRB-

The

contrast

of

the

eigenvector

in

each

stimulus

is

measured

as

the

dot

product

of

the

eigenvector

and

the

stimulus

and

the

contrast

response

function

for

the

eigenvector

is

computed

as

the

average

firing

rate

of

the

cell

at

each

contrast

of

the

eigenvector

For

all

complex

cells

the

firing

rate

was

found

to

increase

with

eigenvector

contrast

at

both

positive

and

negative

polarities

-LRB-

Fig.

-RRB-

consistent

with

the

known

polarity

invariance

of

complex

cells

-LRB-

Hubel

and

Wiesel

-RRB-

The

bimodal

contrast

response

functions

together

with

the

spatial

phase

relationship

between

the

pair

of

significant

eigenvectors

for

each

cell

are

consistent

with

the

well

known

energy

Figure

Models

of

complex

cells

recovered

from

a

covariance

analysis

A

Experimental

protocol

Top

A

segment

of

the

natural

image

ensemble

white

box

indicates

area

shown

in

experiments

Bottom

Spike

train

Spike-triggered

ensemble

was

generated

by

collecting

the

image

preceding

each

spike

by

a

single

frame

-LRB-

ms

-RRB-

B

Two

significant

eigenvectors

of

a

complex

cell

Scale

bar

Solid

line

Spatial

profiles

of

each

eigenvector

along

the

axis

perpendicular

to

the

preferred

orientation

Dashed

line

Gabor

fit

The

Gabor

fits

of

the

two

eigenvectors

had

a

phase

difference

of

C

Contrast

response

functions

of

the

two

eigenvectors

Average

firing

rate

is

plotted

against

the

contrast

of

each

eigenvector

shown

in

B.

Error

bar

x

indicates

SEM

Dashed

lines

Fits

of

the

data

with

the

function

r

-LRB-

x

-RRB-

where

r

is

the

firing

rate

x

is

contrast

and

and

are

free

parameters

D

E

Prediction

of

cortical

responses

to

natural

images

Correlation

coefficients

between

the

predicted

and

measured

responses

based

on

the

eigenvectors

were

plotted

again

those

based

on

the

linear

receptive

field

-LRB-

D

-RRB-

and

against

the

estimated

upper

bound

-LRB-

E

-RRB-

Each

symbol

represents

one

complex

cell

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

model

for

complex

cells

-LRB-

Adelson

and

Bergen

-RRB-

The

energy

model

combines

the

outputs

of

a

quadrature

pair

of

subunits

to

produce

an

orientation-selective

and

spatial

frequency-selective

but

phase-invariant

complex-cell

receptive

field

-LRB-

Fig.

-RRB-

Mathematically

the

model

can

be

described

as

R

F

-LRB-

k

s

-RRB-

F

-LRB-

k

s

-RRB-

where

R

is

the

response

of

the

neuron

s

is

the

stimulus

k

is

the

receptive

field

of

a

subunit

-LRB-

represents

preferred

spatial

phase

-RRB-

and

F

represents

the

contrast

response

function

of

the

subunit

which

is

commonly

approximated

by

F

-LRB-

x

-RRB-

x

The

pair

of

significant

eigenvectors

identified

in

the

STC

analysis

appear

to

correspond

nicely

to

k

and

k

Note

that

the

quadratic

nonlinearity

of

the

energy

model

in

fact

provides

the

perfect

substrate

for

the

STC

analysis

which

computes

the

second-order

Wiener

kernel

of

the

cell

Thus

the

consistency

between

the

STC

result

and

the

energy

model

is

not

entirely

surprising

Interestingly

however

artificial

neural

networks

that

are

trained

to

approximate

the

input

output

relationship

of

complex

cells

also

tend

to

converge

to

connection

profiles

that

closely

resemble

the

energy

model

-LRB-

Lau

et

al.

Prenger

et

al.

-RRB-

suggesting

that

given

the

structural

constraint

of

feedforward

neural

networks

the

energy

model

is

especially

suited

for

approximating

the

responses

of

complex

cells

Although

the

STC

analysis

on

the

complex

cells

of

the

cat

has

yielded

results

consistent

with

the

energy

model

the

result

in

monkey

is

more

complicated

In

the

monkey

STC

analysis

consistently

revealed

additional

excitatory

and

even

suppressive

eigenvectors

for

some

complex

cells

suggesting

the

existence

of

excitatory

and

suppressive

influences

beyond

those

predicted

by

the

energy

model

-LRB-

Rust

et

al.

-RRB-

The

structure

of

the

additional

excitatory

eigenvectors

was

consistent

with

a

model

in

which

complex

cells

are

constructed

by

the

convergence

of

a

number

of

spatially

shifted

subunits

The

suppressive

eigenvectors

had

a

primarily

divisive

influence

on

the

excitatory

response

and

was

consistent

with

a

weighted

normalization

mechanism

The

difference

between

the

results

in

cat

and

monkey

may

be

attributed

to

the

species

difference

or

to

certain

technical

factors

such

as

the

amount

of

data

available

for

the

analysis

The

studies

described

above

have

led

to

relatively

compact

descriptions

of

complex-cell

receptive

fields

Based

on

these

descriptions

how

much

can

we

predict

the

responses

of

the

neuron

to

arbitrary

stimuli

Here

we

will

first

consider

the

responses

to

simple

stimuli

that

are

commonly

used

to

measure

the

response

properties

of

visual

neurons

We

will

then

address

the

issue

of

complex

stimuli

including

natural

scenes

Direction

selectivity

In

studies

using

random-bar

stimuli

the

subunit

receptive

fields

of

many

complex

cells

consist

of

ON

and

OFF

subregions

shifting

smoothly

over

time

-LRB-

spatiotemporally

inseparable

receptive

field

-RRB-

suggesting

direction

selectivity

-LRB-

Lau

et

al.

Rust

et

al.

-RRB-

Previous

studies

have

shown

that

direction

selectivity

of

simple

cells

can

be

predicted

from

their

spatiotemporal

receptive

fields

and

the

expansive

nonlinearity

in

the

contrast

response

function

-LRB-

Albrecht

and

Geisler

DeAngelis

et

al.

Heeger

-RRB-

Because

the

responses

of

complex

cells

are

approximated

as

the

sum

of

two

or

more

subunits

one

can

predict

their

direction

selectivity

using

the

receptive

field

of

each

subunit

and

its

contrast

response

function

Lau

et

al.

-LRB-

-RRB-

made

the

prediction

based

on

the

subunit

receptive

fields

identified

with

neural

networks

and

found

that

the

prediction

agreed

reasonably

well

with

the

direction

selectivity

measured

with

drifting

sinusoidal

gratings

The

correlation

coefficient

between

the

predicted

and

actual

direction

selectivity

was

which

is

considerably

higher

than

that

based

on

the

linear

receptive

field

estimated

by

simple

spike-triggered

averaging

-LRB-

correlation

coefficient

r

-RRB-

Of

course

this

prediction

is

not

perfect

and

it

is

unclear

yet

how

much

the

prediction

error

is

attributable

to

errors

in

the

estimated

model

parameters

and

how

much

is

attributable

to

other

types

of

nonlinearities

not

captured

by

the

model

Orientation

selectivity

When

complex-cell

receptive

fields

are

mapped

with

twodimensional

stimuli

the

spatial

structure

of

their

subunits

clearly

suggests

orientation

selectivity

-LRB-

Fig.

B

-RRB-

One

can

again

predict

the

orientation

tuning

of

the

cell

based

on

the

subunit

receptive

fields

and

their

contrast

response

functions

In

a

study

in

cat

-LRB-

Touryan

et

al.

-RRB-

the

predicted

tuning

curve

agreed

well

with

the

tuning

curve

measured

with

drifting

gratings

in

both

the

preferred

orientation

-LRB-

absolute

difference

-RRB-

and

the

tuning

bandwidth

-LRB-

mean

absolute

difference

-RRB-

The

dot

product

between

the

predicted

and

the

measured

tuning

curves

was

-LRB-

SD

-RRB-

Note

that

in

this

prediction

the

nonlinear

transformation

from

intracellular

signal

to

spike

output

is

reflected

in

the

contrast

response

function

-LRB-

Fig.

-RRB-

which

helps

to

prevent

overestimation

of

the

orientation

tuning

bandwidth

-LRB-

see

above

Direction

selectivity

-RRB-

Spatial

frequency

tuning

The

subunit

receptive

fields

can

also

be

used

to

derive

the

spatial

frequency

tuning

which

was

tested

in

the

STC

analysis

in

cat

-LRB-

Touryan

et

al.

-RRB-

However

the

result

is

confounded

by

the

fact

that

the

receptive

fields

were

mapped

with

natural

images

which

requires

a

modification

of

the

STC

method

that

corrects

for

the

spatial

correlations

in

natural

images

Because

the

stimulus

power

at

high

spatial

frequencies

is

relatively

low

for

natural

images

-LRB-

Field

Dong

and

Atick

Simoncelli

and

Olshausen

-RRB-

this

correction

tends

to

amplify

noise

in

the

high-frequency

range

-LRB-

Theunissen

et

al.

-RRB-

To

reduce

noise

amplification

the

correction

is

made

only

at

spatial

frequencies

below

a

cutoff

point

which

directly

affects

the

predicted

spatial

frequency

tuning

of

the

cell

Thus

how

well

the

complexcell

receptive

fields

measured

with

STC

can

predict

the

spatial

frequency

tuning

of

the

cells

remains

to

be

examined

Complex

stimuli

As

an

ultimate

validation

of

the

STC

model

for

complex

cells

the

model

is

used

to

predict

the

responses

to

complex

stimulus

ensembles

including

random

bars

and

natural

images

following

the

same

procedure

as

that

used

for

predicting

direction

and

orientation

tuning

To

compare

the

predicted

and

measured

firing

rates

an

important

limiting

factor

is

the

variability

in

the

measured

firing

rate

attributable

to

the

limited

number

of

repeats

of

the

stimulus

sequence

The

upper

bound

of

the

correlation

coefficient

between

the

predicted

and

measured

signals

attributable

to

measurement

variability

can

be

estimated

from

the

measured

responses

-LRB-

Hsu

et

al.

-RRB-

When

the

correlation

coefficient

between

the

predicted

and

the

measured

responses

is

plotted

against

this

estimated

upper

bound

it

is

clear

that

although

the

prediction

based

on

the

STC

model

is

consistently

better

than

that

based

on

simple

spike-triggered

averaging

-LRB-

Fig.

D

-RRB-

it

is

far

below

the

upper

bound

for

the

majority

of

complex

cells

-LRB-

Fig.

E

-RRB-

There

may

be

several

factors

that

affect

the

performance

of

the

model

First

there

is

likely

to

be

inaccuracy

of

the

model

because

of

limited

amount

of

data

For

example

there

may

be

additional

subunits

that

are

not

identified

by

the

STC

analysis

-LRB-

Rust

et

al.

-RRB-

The

accuracy

of

the

measured

eigenvectors

and

their

con

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

Figure

Spatial

receptive

fields

of

four

neurons

estimated

using

dynamic

grating

sequences

-LRB-

top

row

-RRB-

and

dynamic

sequences

of

natural

images

-LRB-

bottom

row

-RRB-

The

spatial

receptive

field

describes

selectivity

in

terms

of

a

joint

orientation-spatial

frequency

tuning

surface

Each

point

in

a

spatial

receptive

field

map

describes

the

relative

response

to

a

stimulus

of

a

particular

orientation

and

spatial

frequency

-LRB-

angle

about

the

origin

and

distance

from

the

origin

respectively

see

legend

at

right

-RRB-

Red

indicates

orientations

and

spatial

frequencies

that

tended

to

increase

responses

blue

indicates

decreased

responses

Both

gratings

and

natural

images

yield

spatial

receptive

fields

in

which

excitatory

tuning

is

centered

around

a

small

range

of

spatial

frequencies

and

orientations

Inhibitory

tuning

tends

to

be

more

diffuse

and

its

structure

depends

on

the

stimulus

trast

response

functions

also

depends

on

the

amount

of

data

Second

cortical

cells

are

known

to

exhibit

several

other

types

of

nonlinearity

including

contrast

adaptation

-LRB-

Maffei

et

al.

-RRB-

gain

control

-LRB-

Heeger

-RRB-

and

contextual

modulation

by

stimuli

outside

of

the

classical

receptive

field

-LRB-

Fitzpatrick

Freeman

et

al.

-RRB-

These

effects

are

clearly

not

captured

by

the

model

that

sums

the

responses

of

two

subunits

and

they

can

certainly

contribute

to

the

failure

of

the

STC

model

in

predicting

the

responses

to

complex

stimuli

In

the

study

in

cat

although

the

receptive

fields

measured

with

natural

images

were

similar

to

those

measured

with

random

stimuli

natural

images

were

found

to

be

more

effective

for

driving

the

cells

resulting

in

a

higher

gain

in

the

contrast

response

functions

-LRB-

Felsen

et

al.

-RRB-

In

a

study

in

monkey

-LRB-

David

et

al.

-RRB-

the

spatiotemporal

receptive

field

measured

with

a

phase-separated

Fourier

model

depends

significantly

on

the

statistics

of

visual

stimuli

-LRB-

Fig.

-RRB-

-LRB-

see

below

Evaluating

what

we

know

about

and

beyond

-RRB-

Because

these

nonlinear

effects

are

not

yet

described

in

a

compact

and

precise

form

it

remains

a

formidable

challenge

to

build

a

universal

model

for

complex

cells

that

can

predict

their

responses

to

arbitrary

stimulus

ensembles

Evaluating

what

we

know

about

and

beyond

Computational

studies

of

primary

visual

cortex

have

produced

powerful

quantitative

models

that

accurately

describe

neuronal

responses

to

simple

stimuli

-LRB-

Daugman

Adelson

and

Bergen

Carandini

et

al.

-RRB-

Recent

experiments

suggest

that

under

natural

viewing

conditions

neuronal

responses

may

deviate

markedly

from

the

predictions

of

established

models

-LRB-

Smyth

et

al.

David

et

al.

Touryan

et

al.

-RRB-

Are

these

deviations

functionally

important

This

section

describes

a

nonlinear

system

identification

-LRB-

NLSI

-RRB-

approach

that

can

address

this

problem

This

approach

has

already

been

used

to

determine

precisely

how

well

current

models

predict

neuronal

responses

during

natural

vision

-LRB-

Theunissen

et

al.

David

et

al.

Prenger

et

al.

David

and

Gallant

in

press

-RRB-

It

can

also

be

used

to

identify

the

underlying

causes

of

poor

predictions

and

to

determine

their

relative

importance

Most

quantitative

neuronal

models

of

visual

processing

are

based

on

neurophysiological

data

gathered

using

simple

stimuli

such

as

bars

gratings

and

spots

of

light

Simple

stimulus

experiments

provide

a

powerful

means

to

test

specific

hypotheses

about

neuronal

function

However

if

neurons

function

differ

ently

under

natural

viewing

conditions

then

models

based

on

simple

stimulus

experiments

might

not

be

accurate

To

determine

how

well

such

models

account

for

natural

visual

responses

they

must

be

used

to

make

quantitative

predictions

about

natural

visual

responses

Then

these

predictions

must

be

tested

in

neurophysiological

experiments

under

conditions

approximating

natural

vision

It

is

therefore

essential

to

have

an

objective

method

for

comparing

visual

models

NLSI

provides

a

general

and

efficient

method

for

testing

the

predictions

of

neuronal

models

under

natural

viewing

conditions

-LRB-

Theunissen

et

al.

-RRB-

Within

the

NLSI

framework

each

neuron

is

a

nonlinear

filter

that

operates

on

the

sensory

input

to

produce

an

output

-LRB-

a

series

of

action

potentials

-RRB-

The

filter

is

estimated

from

neurophysiological

data

and

expressed

as

an

equation

that

describes

the

functional

response

properties

of

the

neuron

NLSI

analysis

requires

several

steps

model

specification

fitting

and

regularization

and

validation

and

evaluation

First

a

computational

model

is

specified

that

describes

the

range

of

potential

filtering

operations

that

a

neuron

might

implement

In

principle

one

can

use

any

quantitative

model

that

instantiates

a

specific

theory

of

neuronal

function

or

a

nonparametric

model

estimated

directly

from

the

data

In

practice

it

is

common

to

use

a

very

simple

and

general

model

that

is

efficient

to

estimate

Second

a

regression

procedure

is

used

to

fit

the

model

to

neurophysiological

data

Many

fitting

algorithms

are

available

the

optimal

algorithm

for

a

specific

problem

depends

on

several

factors

including

the

complexity

of

the

stimulus

the

stimulus

and

response

sample

sizes

neuronal

noise

and

the

nature

of

the

computational

model

In

neurophysiological

experiments

on

natural

vision

a

successful

fit

will

depend

critically

on

the

regularization

procedure

used

to

reduce

the

influence

of

neuronal

noise

-LRB-

Theunissen

et

al.

-RRB-

The

third

step

of

NLSI

is

validation

and

evaluation

of

model

predictions

Cortical

neurons

are

quite

variable

in

their

responses

and

this

variability

will

tend

to

cause

overfitting

Models

that

have

many

free

parameters

will

tend

to

fit

both

systematic

variability

and

experimental

noise

This

overfitting

problem

makes

it

difficult

to

compare

models

with

different

numbers

of

free

parameters

However

this

problem

can

be

eliminated

by

using

a

strict

cross-validation

procedure

-LRB-

Theunissen

et

al.

David

et

al.

David

and

Gallant

in

press

-RRB-

Before

analysis

a

portion

of

the

data

-LRB-

typically

-RRB-

is

set

aside

for

validation

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

The

model

is

then

fit

to

the

remaining

data

Performance

of

the

model

is

assessed

by

evaluating

its

ability

to

predict

responses

in

the

reserved

validation

dataset

The

performance

of

a

model

is

usually

expressed

as

the

correlation

between

predicted

and

observed

responses

in

the

validation

dataset

-LRB-

Theunissen

et

al.

David

et

al.

David

and

Gallant

in

press

-RRB-

although

other

measures

have

also

been

used

-LRB-

Hsu

et

al.

-RRB-

One

important

consideration

when

evaluating

such

correlations

is

the

intrinsic

variability

of

the

data

Neuronal

noise

and

the

data

sample

size

place

an

upper

limit

on

the

correlation

between

any

model

and

a

specific

dataset

-LRB-

Hsu

et

al.

David

and

Gallant

in

press

-RRB-

In

any

real

experiment

even

the

best

possible

model

can

not

produce

a

perfect

correlation

between

predicted

and

observed

responses

The

potentially

explainable

variance

is

the

fraction

of

total

response

variance

that

could

theoretically

be

predicted

given

neuronal

noise

and

a

finite

sample

The

predictive

power

of

a

model

is

the

percentage

of

potentially

explainable

variance

that

is

in

fact

explained

by

the

model

Although

the

NLSI

approach

has

been

used

in

neurophysiology

for

several

decades

early

studies

only

used

white

noise

or

simple

stimuli

with

flat

power

spectra

-LRB-

DeBoer

and

Kuyper

Sutter

Emerson

et

al.

Jones

and

Palmer

-RRB-

Because

the

statistics

of

natural

stimuli

are

not

white

-LRB-

Field

Dong

and

Atick

-RRB-

a

nonlinear

neuron

might

respond

differently

to

these

simple

stimuli

than

it

would

to

natural

images

Fortunately

recent

theoretical

and

experimental

studies

have

shown

that

NLSI

can

also

be

used

with

natural

images

-LRB-

Theunissen

et

al.

Willmore

and

Smyth

David

et

al.

David

and

Gallant

in

press

-RRB-

Early

NSLI

studies

usually

used

simple

first

or

second-order

Volterra/Wiener

regression

models

-LRB-

Marmarelis

and

Marmarelis

Emerson

et

al.

Eggermont

-RRB-

These

first

and

second-order

polynomial

models

are

computationally

tractable

and

simple

to

fit

However

more

complicated

models

that

account

for

nonlinear

mechanisms

such

as

contrast

gain

control

-LRB-

Carandini

et

al.

-RRB-

and

nonclassical

receptive-field

modulation

-LRB-

Knierim

and

van

Essen

Vinje

and

Gallant

-RRB-

should

in

theory

perform

better

than

a

simple

first

or

secondorder

model

Several

studies

have

now

used

some

variant

of

the

NLSI

approach

to

investigate

how

cortical

neurons

encode

natural

images

-LRB-

Lehky

et

al.

Theunissen

et

al.

Ringach

et

al.

Smyth

et

al.

Willmore

and

Smyth

David

et

al.

Prenger

et

al.

David

and

Gallant

Touryan

et

al.

-RRB-

David

et

al.

-LRB-

-RRB-

evaluated

the

performance

of

current

models

by

examining

their

ability

to

predict

natural

visual

responses

The

stimuli

were

movies

that

simulated

the

spatial

and

temporal

stimulation

that

a

single

neuron

would

receive

during

free

inspection

of

a

static

monochromatic

natural

scene

Neurophysiological

recordings

were

made

from

single

neurons

during

presentation

with

these

natural

vision

movies

David

et

al.

-LRB-

-RRB-

developed

a

spatiotemporal

phase-separated

Fourier

model

-LRB-

PSFT

-RRB-

model

that

can

account

for

many

of

the

known

properties

of

neurons

According

to

this

model

the

spatial

response

of

any

neuron

is

given

by

the

weighted

linear

sum

of

stimulus

energy

across

orientation

spatial

frequency

and

phase

channels

followed

by

an

output

nonlinearity

The

temporal

responses

are

modeled

as

a

simple

linear

filter

Because

the

PSFT

model

incorporates

both

first

and

second-order

terms

it

can

account

for

both

the

phase-sensitive

responses

of

simple

cells

and

the

phase-independent

responses

of

complex

cells

-LRB-

De

Valois

et

al.

-RRB-

The

model

describes

both

excitatory

and

inhibitory

interactions

between

spatial

frequency

channels

and

it

can

ap

proximate

neuronal

mechanisms

mediating

contrast

gain

control

-LRB-

Carandini

et

al.

-RRB-

and

nonclassical

modulation

-LRB-

Knierim

and

van

Essen

Vinje

and

Gallant

-RRB-

Finally

the

PSFT

can

capture

the

second-order

mechanisms

revealed

by

the

Wiener

-LRB-

Emerson

et

al.

-RRB-

and

spike-triggered

covariance

methods

-LRB-

see

above

Understanding

complex

cells

-RRB-

-LRB-

Rust

et

al.

Touryan

et

al.

-RRB-

David

et

al.

-LRB-

-RRB-

used

a

linearized

reverse-correlation

procedure

to

fit

the

PSFT

model

to

the

data

acquired

from

each

neuron

-LRB-

a

software

toolbox

for

conducting

this

analysis

is

available

at

-RRB-

They

found

that

spatiotemporal

receptive

fields

estimated

using

the

PSFT

model

predict

of

the

total

variance

in

responses

of

neurons

during

simulated

natural

vision

In

a

later

study

-LRB-

David

and

Gallant

-RRB-

they

measured

the

intrinsic

noise

level

in

these

experiments

to

determine

explainable

variance

and

predictive

power

They

found

that

a

second-order

Fourier

power

-LRB-

FP

-RRB-

model

-LRB-

see

below

-RRB-

predicts

of

the

explainable

variance

in

the

responses

of

neurons

during

natural

vision

This

figure

represents

the

best

current

estimate

of

how

well

conventional

models

account

for

natural

visual

responses

The

study

by

David

et

al.

-LRB-

-RRB-

also

included

two

additional

stimulus

classes

rapid

sequences

of

random

gratings

that

had

white

spatial

and

temporal

statistics

and

rapid

sequences

of

natural

images

that

had

the

spatial

statistics

of

natural

vision

movies

and

the

white

temporal

statistics

of

grating

sequences

A

separate

spatiotemporal

receptive

field

was

estimated

from

the

responses

to

each

stimulus

class

giving

three

-LRB-

potentially

different

-RRB-

filter

estimates

for

each

neuron

They

reported

that

receptive

field

generated

using

data

from

a

single

stimulus

class

predict

responses

within

that

stimulus

class

better

than

across

classes

Temporal

stimulus

statistics

have

a

large

effect

on

estimated

receptive

field

temporal

responses

become

more

bimodal

and

short-term

adaptation

increases

as

the

temporal

stimulus

statistics

become

more

natural

-LRB-

less

white

-RRB-

Spatial

stimulus

statistics

have

a

more

subtle

effect

on

the

receptive

field

excitatory

tuning

appears

to

be

stable

regardless

of

the

spatial

stimulus

structure

but

natural

spatial

statistics

evoke

complex

changes

in

the

pattern

of

inhibitory

tuning

Together

these

observations

demonstrate

that

the

functional

properties

of

neurons

depend

partly

on

the

prevailing

stimulus

statistics

-LRB-

Fig.

-RRB-

David

et

al.

-LRB-

-RRB-

reported

that

the

modest

performance

of

the

PSFT

model

was

most

likely

attributable

to

the

unanticipated

influence

of

temporal

stimulus

statistics

on

response

properties

Temporal

stimulus

statistics

dramatically

change

the

temporal

integration

properties

of

these

neurons

and

a

linear

temporal

filter

can

not

account

for

this

temporal

nonlinearity

Current

research

-LRB-

J.

L.

Gallant

unpublished

observations

-RRB-

suggests

that

a

more

sophisticated

temporally

nonlinear

model

will

substantially

improve

predictions

To

improve

model

performance

beyond

this

will

require

a

still

more

complicated

model

that

explicitly

represents

other

nonlinear

mechanisms

such

as

the

spatial

interaction

between

the

classical

and

nonclassical

receptive

field

-LRB-

Knierim

and

van

Essen

Vinje

and

Gallant

-RRB-

Most

previous

NLSI

studies

have

focused

on

the

peripheral

nervous

system

or

primary

sensory

cortex

work

on

extrastriate

visual

areas

such

as

is

in

its

infancy

One

current

study

-LRB-

Gallant

unpublished

observations

-RRB-

uses

a

quantitative

FP

model

that

describes

orientation

tuning

spatial

frequency

tuning

and

position

invariance

reported

previously

in

-LRB-

Desimone

and

Schein

Gallant

et

al.

-RRB-

Recordings

were

made

from

single

neurons

during

stimulation

with

a

Hz

sequence

of

natural

images

-LRB-

Hayden

and

Gallant

-RRB-

A

linearized

reverse

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

correlation

procedure

was

used

to

fit

the

FP

model

to

the

data

acquired

from

each

neuron

The

estimated

spatial

receptive

fields

predict

on

average

of

the

explainable

variance

of

spatial

responses

in

-LRB-

Gallant

unpublished

observations

-RRB-

These

predictions

are

substantially

worse

than

those

obtained

in

area

although

the

study

did

not

attempt

to

model

the

temporal

receptive

field

This

result

confirms

that

relatively

little

is

currently

known

about

the

functional

properties

of

neurons

in

extrastriate

visual

areas

along

the

ventral

pathway

The

predictive

power

of

any

neuronal

model

will

depend

on

how

accurately

the

model

captures

the

nonlinear

stimulus

response

mapping

function

of

a

neuron

Because

there

are

currently

no

commonly

accepted

models

of

processing

in

it

is

not

surprising

that

the

FP

model

performs

poorly

One

strategy

for

developing

an

appropriate

model

is

to

use

NLSI

to

generate

the

model

itself

These

nonparametric

models

are

not

specified

explicitly

but

instead

emerge

from

the

data

during

the

fitting

procedure

Several

nonparametric

models

have

been

used

to

characterize

neurons

artificial

neural

networks

-LRB-

Lehky

et

al.

Lau

et

al.

Prenger

et

al.

-RRB-

maximally

informative

dimensions

-LRB-

Sharpee

et

al.

-RRB-

and

kernel

regression

methods

such

as

the

support

vector

machine

-LRB-

Wu

and

Gallant

-RRB-

Because

these

models

do

not

require

any

previous

theory

about

the

nature

of

neural

coding

they

are

also

likely

to

be

useful

for

characterizing

neurons

in

visual

areas

beyond

One

complication

of

nonparametric

models

is

that

the

estimated

spatiotemporal

receptive

field

may

be

broadly

distributed

across

many

coefficients

in

such

a

way

that

it

can

not

be

interpreted

directly

In

such

cases

a

separate

visualization

procedure

can

be

used

to

interpret

the

receptive

field

-LRB-

Lau

et

al.

Prenger

et

al.

-RRB-

Responses

in

extrastriate

areas

such

as

are

strongly

modulated

by

attention

-LRB-

Connor

et

al.

Gallant

-RRB-

but

most

neurophysiological

studies

of

attentional

modulation

are

conducted

under

conditions

much

simpler

than

those

prevailing

during

natural

vision

One

recent

study

-LRB-

David

et

al.

-RRB-

investigated

how

attention

modulates

neuronal

responses

in

area

during

a

naturalistic

free-viewing

visual

search

task

-LRB-

Mazer

and

Gallant

-RRB-

The

FP

model

and

linearized

reverse

correlation

were

used

to

estimate

the

spatial

receptive

field

of

each

neuron

A

stepwise

regression

procedure

was

then

used

to

identify

attentional

modulation

of

the

mean

rate

response

gain

and

orientation

and

spatial

frequency

tuning

David

et

al.

-LRB-

-RRB-

found

that

attention

modulates

all

three

aspects

of

the

spatial

receptive

field

it

changes

mean

response

rate

and

gain

and

it

modulates

orientation

and

spatial

frequency

tuning

Additional

investigation

of

more

naturalistic

experimental

conditions

will

be

crucial

to

obtain

an

accurate

understanding

the

role

of

attention

during

natural

vision

The

ultimate

test

of

any

theory

of

the

neural

basis

of

visual

perception

is

its

ability

to

predict

neuronal

responses

during

natural

vision

NLSI

provides

an

efficient

framework

for

developing

testable

models

fitting

them

to

neurophysiological

data

and

evaluating

their

predictive

power

This

process

also

provides

a

means

to

compare

models

objectively

based

on

their

ability

to

predict

responses

during

natural

vision

Such

comparisons

can

facilitate

model

selection

and

identify

promising

areas

for

future

research

The

available

evidence

-LRB-

David

et

al.

David

and

Gallant

-RRB-

indicates

that

current

models

predict

of

the

explainable

variance

in

responses

of

neurons

during

natural

vision

Preliminary

data

suggest

that

there

remains

much

to

learn

about

the

functional

characteristics

of

neurons

beyond

In

extrastriate

visual

areas

such

as

most

of

the

knowable

is

still

unknown

What

we

do

n't

know

about

The

past

years

have

produced

enormous

amounts

of

data

concerning

the

structure

and

function

of

area

in

a

variety

of

different

species

of

mammals

What

has

emerged

from

this

work

among

other

things

is

a

fairly

well

agreed

on

standard

model

of

neuron

response

properties

usually

involving

a

combination

of

linear

filtering

half-wave

rectification

and

squaring

and

response

normalization

-LRB-

described

above

in

previous

sections

-RRB-

Although

this

model

is

well

supported

by

much

of

the

available

data

it

is

still

unknown

how

well

it

fares

in

accounting

for

the

actual

behavior

of

the

entire

population

of

neurons

when

presented

with

the

full

complexity

of

time-varying

natural

scenes

In

previous

work

Olshausen

and

Field

-LRB-

-RRB-

attempted

to

quantify

our

current

level

of

understanding

of

function

by

considering

two

important

factors

an

estimate

of

the

fraction

of

neuron

types

that

are

typically

characterized

in

experimental

studies

and

the

fraction

of

variance

explained

in

the

responses

of

these

neurons

under

natural

viewing

conditions

Together

these

two

factors

led

them

to

conclude

that

at

present

as

much

as

of

function

has

yet

to

be

accounted

for

They

identified

five

specific

problems

that

will

need

to

be

overcome

before

we

can

claim

to

have

an

accurate

standard

model

of

function

These

are

briefly

reviewed

here

Biased

sampling

of

neurons

The

vast

majority

of

our

knowledge

about

function

has

been

obtained

from

single-unit

recordings

in

which

a

microelectrode

is

brought

into

close

proximity

with

a

neuron

in

cortex

Ideally

when

doing

this

one

would

like

to

obtain

an

unbiased

sample

from

any

given

layer

of

cortex

but

some

biases

are

difficult

to

avoid

The

most

troubling

of

these

is

that

the

process

of

hunting

for

neurons

with

a

single

microelectrode

will

typically

steer

one

toward

neurons

with

higher

firing

rates

Recent

studies

of

energy

consumption

by

neurons

estimate

that

the

average

activity

of

neurons

be

relatively

low

i.e.

spike/s

in

primate

cortex

-LRB-

Attwell

and

Laughlin

Lennie

-RRB-

However

one

finds

many

studies

in

the

literature

in

which

even

the

spontaneous

or

background

rates

are

well

above

spike/s

suggesting

that

the

more

active

neurons

are

substantially

overrepresented

-LRB-

Lennie

-RRB-

Olshausen

and

Field

-LRB-

-RRB-

estimate

that

as

much

as

of

the

population

of

neurons

in

may

have

been

missed

because

of

this

bias

A

number

of

recent

studies

show

that

when

one

searches

for

neurons

using

less

biased

methods

such

as

chronic

implants

or

antidromic

stimulation

neurons

with

substantially

lower

firing

rates

become

much

more

common

-LRB-

for

review

see

Olshausen

and

Field

-RRB-

If

the

same

is

true

for

then

we

will

need

to

characterize

the

response

properties

of

such

neurons

and

if

they

are

substantially

different

we

may

need

to

revisit

our

beliefs

about

how

typical

neurons

behave

Biased

stimuli

Many

elements

of

the

existing

standard

model

for

neurons

were

derived

from

experiments

using

a

fairly

restricted

class

of

test

stimuli

Oftentimes

these

stimuli

are

ideal

for

characterizing

linear

systems

i.e.

spots

white

noise

or

sine-wave

gratings

or

else

they

are

designed

around

preexisting

notions

of

how

neurons

should

respond

The

hope

is

that

the

insights

gained

from

studying

neurons

using

these

reduced

stimuli

will

generalize

to

more

complex

situations

i.e.

natural

scenes

However

in

a

nonlinear

system

the

response

to

any

reduced

set

of

stimuli

can

not

be

guaranteed

to

provide

the

information

needed

to

predict

the

response

to

an

arbitrary

combination

of

those

stimuli

The

extent

to

which

this

holds

for

has

yet

to

be

thoroughly

examined

Be

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

cause

it

is

impossible

to

map

out

the

response

to

all

possible

stimuli

some

assumptions

about

the

nature

of

the

nonlinearity

and

the

stimulus

space

must

be

made

The

assumption

that

Olshavsen

and

Field

-LRB-

-RRB-

believe

is

appropriate

is

that

the

nonlinearities

relevant

to

visual

processing

are

most

likely

to

be

revealed

when

the

system

is

presented

with

ecologically

relevant

stimuli

Traditionally

experimentalists

have

been

reluctant

to

use

natural

scenes

as

stimuli

because

they

seem

highly

variable

and

uncontrolled

However

in

recent

years

there

has

been

significant

progress

in

modeling

the

structure

of

natural

images

-LRB-

Simoncelli

and

Olshausen

-RRB-

and

it

should

soon

be

possible

to

develop

parametric

descriptions

of

natural

images

that

could

be

used

to

generate

experimental

stimuli

-LRB-

Heeger

and

Bergen

-RRB-

Several

recently

developed

adaptive

stimulus

techniques

also

provide

a

promising

avenue

for

determining

the

relevant

stimulus

for

sensory

neurons

-LRB-

Edin

et

al.

Foldiak

et

al.

O'Connor

et

al.

-RRB-

Biased

theories

Currently

in

neuroscience

there

is

an

emphasis

on

telling

a

story

This

often

encourages

investigators

to

demonstrate

when

a

theory

explains

data

not

when

a

theory

provides

a

poor

model

In

addition

editorial

pressures

can

encourage

one

to

make

a

tidy

picture

out

of

data

that

may

actually

be

quite

messy

The

result

is

that

theories

emerge

that

are

centered

around

explaining

a

particular

subset

of

published

data

or

that

can

be

conveniently

proven

rather

than

being

motivated

by

functional

considerations

A

good

theory

should

not

only

explain

data

but

it

must

also

address

how

the

problems

of

vision

are

solved

by

the

cortex

For

example

much

of

our

thinking

about

function

has

been

guided

by

the

notion

that

there

are

two

distinct

classes

of

cells

simple

and

complex

but

a

number

of

recent

studies

are

now

calling

this

into

question

pointing

out

that

this

classification

scheme

could

simply

be

an

artifact

of

the

lens

through

which

we

view

the

data

-LRB-

Mechler

and

Ringach

Priebe

et

al.

-RRB-

Indeed

given

the

variety

of

response

properties

one

observes

it

can

become

quite

difficult

to

shoehorn

any

given

cell

into

one

of

these

two

categories

-LRB-

see

above

Understanding

simple

cells

-RRB-

Another

theory

bias

often

embedded

in

investigations

of

function

is

the

notion

that

simple

cells

and

complex

cells

are

actually

coding

for

the

presence

of

edges

corners

or

other

twodimensional

shape

features

in

images

However

despite

much

effort

in

computer

vision

it

has

proven

impossible

to

detect

even

the

simple

outline

of

an

object

using

a

filter

such

as

a

simple

or

complex-cell

model

Moreover

it

is

entirely

unclear

whether

such

a

representation

would

be

meaningful

or

useful

in

the

first

place

One

of

the

most

challenging

problems

facing

the

cortex

is

that

of

inferring

a

representation

of

threedimensional

surfaces

from

the

two-dimensional

image

-LRB-

Nakayama

et

al.

-RRB-

This

is

not

an

easy

problem

to

solve

and

it

still

lies

beyond

the

capabilities

of

modern

computer

vision

It

seems

quite

likely

that

plays

a

role

in

solving

this

problem

but

understanding

how

it

does

so

might

require

going

beyond

bottom-up

filtering

models

to

consider

how

top-down

information

is

used

in

the

interpretation

of

images

-LRB-

Lee

and

Mumford

-RRB-

Interdependence

and

contextual

effects

A

combination

of

anatomical

and

physiological

studies

suggest

that

of

the

response

variance

of

a

layer

neuron

is

a

function

of

other

neurons

or

inputs

other

than

those

arising

from

LGN

-LRB-

for

review

see

Olshausen

and

Field

-RRB-

Determining

how

these

contextual

signals

influence

the

response

prop

erties

of

neurons

has

been

the

subject

of

many

investigations

over

the

past

decade

often

using

bars

or

gratings

to

probe

how

stimuli

in

the

surround

affect

the

response

to

a

stimulus

in

the

center

of

the

receptive

field

-LRB-

for

review

see

Albright

and

Stoner

Series

et

al.

-RRB-

However

the

problem

one

faces

in

teasing

apart

contextual

effects

this

way

is

the

combinatorial

explosion

in

exploring

all

of

the

possible

spatial

and

featural

configurations

of

surrounding

stimuli

What

we

really

want

to

know

is

how

neurons

respond

within

the

sorts

of

context

encountered

in

natural

scenes

Some

of

the

initial

studies

exploring

the

role

of

context

in

natural

scenes

have

demonstrated

pronounced

nonlinear

effects

that

tend

to

sparsify

activity

in

a

way

that

would

have

been

hard

to

predict

from

the

existing

studies

-LRB-

Vinje

and

Gallant

-RRB-

More

studies

along

these

lines

are

needed

and

most

importantly

we

need

to

understand

how

and

why

the

context

in

natural

scenes

produces

such

effects

Ecological

deviance

In

the

past

few

years

a

number

of

laboratories

have

begun

using

natural

scenes

as

stimuli

when

recording

from

neurons

in

the

visual

pathway

In

particular

Gallant

and

collaborators

have

taken

the

approach

of

attempting

to

determine

how

well

one

can

predict

the

responses

of

neurons

to

natural

stimuli

using

a

variety

of

different

models

For

example

David

et

al.

-LRB-

-RRB-

have

explored

two

different

types

of

models

a

linearized

spatiotemporal

receptive

field

model

in

which

the

response

of

the

neuron

is

essentially

a

weighted

sum

of

the

image

pixels

over

space

and

time

and

a

phase-separated

Fourier

model

that

allows

one

to

capture

the

phase-invariance

nonlinearity

of

a

complex

cell

-LRB-

see

above

Evaluating

what

we

know

about

-RRB-

These

models

can

typically

explain

between

and

of

the

response

variance

of

neurons

One

could

possibly

obtain

a

better

fit

to

the

data

by

including

additional

terms

modeling

suppression

-LRB-

Rust

et

al.

-RRB-

and

temporal

adaptation

-LRB-

Lesica

et

al.

-RRB-

or

even

a

spiking

mechanism

-LRB-

Paninski

et

al.

-RRB-

but

it

is

still

sobering

to

realize

that

the

receptive

field

component

per

se

which

is

the

bread

and

butter

of

the

standard

model

accounts

for

so

little

of

the

response

variance

Moreover

the

way

in

which

these

models

fail

does

not

leave

one

optimistic

that

the

addition

of

modulatory

terms

or

pointwise

nonlinearities

will

fix

matters

Typically

the

model

will

undershoot

the

response

of

the

neuron

but

there

are

also

many

events

in

the

response

that

are

completely

missed

by

the

model

and

vice

versa

This

is

in

stark

contrast

to

the

LGN

in

which

the

linear

model

predicts

nearly

every

event

in

the

response

but

mainly

differs

by

a

gain

factor

that

can

often

be

corrected

by

the

proper

application

of

a

gain

control

mechanism

-LRB-

see

above

Understanding

LGN

responses

-RRB-

-LRB-

Fig.

-RRB-

Also

in

an

experiment

in

ferrets

using

brief

presentations

of

static

natural

images

the

linear

model

often

appears

to

succeed

in

conveying

the

gist

of

neural

responses

-LRB-

Smyth

et

al.

-RRB-

-LRB-

see

above

Understanding

simple

cells

-RRB-

Thus

there

appears

to

be

a

qualitative

mismatch

in

predicting

the

responses

of

cortical

neurons

to

time-varying

natural

images

that

will

require

more

than

tweaking

to

resolve

What

seems

to

be

suggested

by

the

data

is

that

a

more

complex

network

nonlinearity

is

at

work

here

and

that

describing

the

behavior

of

any

one

neuron

will

require

one

to

include

the

influence

of

other

simultaneously

recorded

neurons

Obtaining

a

more

complete

and

accurate

picture

of

function

will

require

three

fundamental

changes

in

our

approach

-LRB-

-RRB-

more

widespread

use

of

recording

techniques

that

allow

for

an

unbiased

sample

of

the

population

of

neurons

across

different

layers

of

cortex

and

their

interactions

-LRB-

-RRB-

the

use

of

time-varying

natural

scenes

or

reasonable

approximations

thereof

for

char

J.

Neurosci.

November

-LRB-

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

acterizing

neural

response

properties

and

-LRB-

-RRB-

the

advancement

of

functionally

driven

and

testable

theories

of

function

The

latter

will

require

that

we

extrapolate

beyond

the

available

data

taking

into

account

the

problems

that

need

to

be

solved

by

the

visual

system

in

addition

to

constraints

provided

by

known

neuroanatomy

and

neurophysiology

Importantly

we

will

need

to

keep

an

open

mind

in

considering

new

theories

and

given

how

little

of

function

we

can

currently

claim

to

understand

we

should

be

prepared

for

some

surprises

as

new

data

come

in

Conclusions

Among

the

views

expressed

in

this

review

there

are

a

number

of

points

of

agreement

but

there

are

also

notable

differences

of

opinion

A

first

point

of

agreement

is

that

an

adequate

model

of

visual

responses

should

predict

responses

to

arbitrary

stimuli

not

only

those

encountered

in

the

laboratory

but

also

those

seen

in

nature

Surprisingly

many

of

the

standard

models

of

early

visual

processing

have

not

been

held

to

this

rigorous

test

As

pointed

out

by

Demb

-LRB-

see

above

Understanding

the

retinal

output

-RRB-

models

of

the

retina

have

not

attempted

to

go

all

the

way

and

predict

responses

to

complex

video

sequences

This

attempt

has

been

made

in

LGN

with

a

linear

model

by

Dan

et

al.

-LRB-

-RRB-

and

with

a

nonlinear

model

described

in

Understanding

LGN

responses

As

described

in

the

subsequent

sections

some

initial

attempts

have

been

made

for

cells

Still

much

work

remains

before

we

can

tell

whether

current

models

are

appropriate

to

predict

responses

to

complex

video

sequences

A

second

point

of

agreement

is

that

a

tractable

model

of

visual

responses

should

include

a

linear

receptive

field

-LRB-

or

more

than

one

in

the

case

of

complex

cells

-RRB-

This

linear

receptive

field

could

operate

directly

on

a

scaled

version

of

the

images

-LRB-

as

in

the

models

derived

from

those

in

Fig.

-RRB-

or

on

a

transformation

of

the

stimulus

such

as

the

Fourier

transform

as

is

the

case

for

the

model

described

by

Gallant

-LRB-

see

above

Evaluating

what

we

know

about

and

beyond

-RRB-

In

fact

even

models

of

complex

cells

that

postulate

highly

nonlinear

image

processing

eventually

combine

the

visual

information

with

a

linear

receptive

field

-LRB-

Ringach

et

al.

-RRB-

Similarly

an

influential

model

of

visual

responses

in

the

cortical

middle

temporal

area

postulates

a

linear

receptive

field

whose

inputs

are

the

outputs

of

neurons

-LRB-

Simoncelli

and

Heeger

-RRB-

A

point

at

which

the

approaches

differ

is

in

the

use

of

complex

stimuli

such

as

natural

video

sequences

One

approach

proposed

above

in

Understanding

LGN

responses

is

to

use

them

only

to

test

a

model

that

has

been

constrained

with

simpler

stimuli

An

alternative

approach

proposed

above

in

Evaluating

what

we

know

about

and

beyond

and

What

we

do

n't

know

about

is

to

use

them

also

to

discover

the

appropriate

type

of

model

and

to

constrain

the

model

There

are

valid

reasons

for

both

approaches

The

first

approach

posits

that

appropriate

models

of

neural

function

are

so

nonlinear

that

it

would

be

hopeless

to

try

to

fit

them

to

responses

to

complicated

stimuli

It

is

better

to

constrain

the

model

with

simpler

stimuli

such

as

gratings

and

then

see

how

the

model

does

with

more

complicated

stimuli

This

approach

has

illustrious

precedent

in

biophysics

for

example

Hodgkin

and

Huxley

characterized

the

mechanisms

of

the

action

potential

by

using

highly

simplified

stimuli

-LRB-

e.g.

by

clamping

the

voltage

of

the

cell

-RRB-

They

did

not

try

to

inject

a

natural-looking

current

that

simulates

the

arrival

of

synaptic

potentials

even

with

the

computers

of

today

this

approach

would

be

unlikely

to

yield

Hodgkin

and

Huxley

's

elegant

equations

However

neurons

in

visual

cortex

and

particularly

in

areas

beyond

are

likely

to

be

specialized

in

scene

analysis

that

goes

well

beyond

the

extraction

of

edges

and

similar

low-level

image

processing

It

could

become

hopeless

to

try

to

characterize

these

neurons

using

simple

stimuli

such

as

gratings

Stimuli

of

that

simplicity

might

perhaps

become

useful

for

this

task

later

after

a

wide

exploration

is

made

with

complex

natural

stimuli

and

the

general

outlines

of

the

mechanisms

underlying

the

responses

have

been

elucidated

Another

point

at

which

there

does

not

seem

to

be

agreement

is

whether

we

have

a

satisfactory

grasp

of

the

computations

performed

in

primary

visual

cortex

Although

no

study

yet

seems

to

have

measured

how

well

a

model

of

retinal

ganglion

cell

would

do

in

predicting

responses

to

a

natural

movie

at

least

for

the

more

linear

X-type

ganglion

cells

there

is

a

sense

that

a

model

that

included

the

key

known

nonlinear

mechanisms

should

do

fairly

well

-LRB-

see

above

Understanding

the

retinal

output

-RRB-

Similarly

the

model

of

LGN

presented

above

-LRB-

see

above

Understanding

LGN

responses

-RRB-

although

far

from

perfect

does

capture

the

gist

of

LGN

responses

to

complex

stimuli

Models

of

neurons

in

visual

cortex

succeed

in

capturing

the

qualitative

-LRB-

tuning

-RRB-

properties

of

these

neurons

but

are

less

robust

at

quantitatively

predicting

the

responses

to

arbitrary

stimulus

sets

Understanding

simple

cells

describes

a

fairly

successful

attempt

made

for

simple

cells

but

that

attempt

only

involved

static

images

The

model

of

complex

cells

described

in

Understanding

complex

cells

can

be

derived

directly

from

responses

to

complex

natural

video

sequences

-LRB-

Touryan

et

al.

-RRB-

but

the

model

does

not

perform

as

well

as

models

of

earlier

visual

neurons

The

measurements

presented

in

Evaluating

what

we

know

about

and

beyond

indicate

that

a

model

that

combines

linear

and

second-order

mechanisms

accounts

for

at

least

one-third

of

the

variance

of

responses

to

complex

stimuli

However

we

do

not

know

how

well

a

more

complete

model

would

do

in

predicting

responses

it

does

not

seem

that

anybody

has

yet

attempted

to

build

a

model

of

neurons

that

includes

many

of

the

known

dynamical

nonlinear

components

-LRB-

e.g.

synaptic

depression

and

surround

suppression

-RRB-

-LRB-

Fig.

-RRB-

and

tested

this

model

on

responses

to

complex

video

sequences

Until

this

work

is

done

the

field

is

open

to

the

criticisms

levied

above

in

What

we

do

n't

know

about

Three

obstacles

lie

in

the

way

of

a

meaningful

comparison

of

different

models

for

a

given

visual

stage

or

across

visual

stages

First

as

long

as

different

models

are

applied

to

responses

of

different

stimuli

-LRB-

flashed

still

pictures

vs

cartoons

vs

natural

video

sequences

-RRB-

we

will

not

be

able

to

compare

model

performance

Second

there

does

not

seem

to

be

agreement

over

the

timescale

of

responses

that

one

is

trying

to

describe

Some

models

attempt

to

predict

responses

down

to

the

individual

spike

-LRB-

Keat

et

al.

Paninski

et

al.

-RRB-

but

more

commonly

one

is

interested

in

firing

rates

computed

in

the

ms

range

-LRB-

see

above

Understanding

simple

cells

-RRB-

Third

there

does

not

seem

to

be

an

agreed

measure

of

model

quality

For

example

the

different

sections

of

this

review

invoked

different

measures

including

correlation

-LRB-

Tolhurst

-RRB-

percentage

of

the

variance

-LRB-

Demb

and

Mante

-RRB-

and

percentage

of

the

explainable

variance

or

explainable

correlation

-LRB-

Gallant

and

Dan

-RRB-

The

last

approaches

seem

the

most

reasonable

because

they

include

an

estimate

of

the

variability

of

responses

which

the

models

are

not

expected

to

capture

Recent

work

indeed

has

yielded

revised

measures

of

percentage

of

variance

-LRB-

Sahani

and

Linden

-RRB-

or

of

correlation

-LRB-

Hsu

et

al.

-RRB-

that

are

adjusted

to

indicate

when

a

model

accounts

for

the

explainable

responses

i.e.

when

the

deviations

from

the

actual

responses

are

within

the

variability

that

is

present

in

the

responses

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

J.

Neurosci.

November

-LRB-

-RRB-

Clearly

much

work

lies

ahead

before

we

can

say

that

we

understand

what

the

early

visual

system

does

Recent

advances

in

stimulus

presentation

and

receptive-field

mapping

techniques

allow

us

for

the

first

time

to

fit

and

test

models

that

can

produce

quantitative

predictions

of

the

response

of

an

individual

neuron

to

large

classes

of

stimuli

Although

these

models

appear

to

perform

reasonably

-LRB-

albeit

imperfectly

-RRB-

in

the

retina

and

LGN

the

performance

of

these

models

degrades

substantially

as

one

ascends

to

the

cortex

Standard

models

of

neurons

in

the

retina

-LRB-

Fig.

A

-RRB-

successfully

describe

both

the

qualitative

characteristics

of

the

tuning

properties

of

the

neuron

-LRB-

e.g.

tuning

for

the

spatial

frequency

of

a

drifting

grating

-RRB-

and

are

reasonable

predictors

of

the

response

of

a

neuron

at

brief

timescales

-LRB-

accounting

for

of

the

variance

in

response

see

above

Understanding

the

retinal

output

-RRB-

Similarly

standard

models

of

simple

cells

-LRB-

Fig.

B

-RRB-

successfully

capture

the

basic

tuning

properties

-LRB-

orientation

direction

and

phase

sensitivity

-RRB-

of

the

most

linear

neurons

falling

in

this

class

In

the

more

nonlinear

simple

and

complex

cells

response

properties

can

no

longer

be

described

by

a

standard

model

including

a

single

linear

filter

and

only

recently

have

new

characterization

techniques

allowed

researchers

to

recover

models

of

these

cells

Models

of

complex

cells

-LRB-

Fig.

-RRB-

quite

successfully

predict

the

qualitative

response

properties

of

these

neurons

such

as

tuning

for

the

orientation

and

direction

of

moving

stimuli

However

standard

models

of

these

neurons

succeed

in

capturing

only

of

the

explainable

variance

in

natural

visual

responses

-LRB-

see

above

Evaluating

what

we

know

about

and

beyond

-RRB-

suggesting

that

crucial

elements

are

missing

from

the

standard

model

description

One

likely

source

of

error

in

standard

models

is

the

absence

of

any

history-dependent

adaptation

At

all

stages

of

processing

visual

neurons

adapt

to

the

recent

luminance

and

contrast

of

stimuli

these

properties

can

not

be

captured

by

the

models

presented

in

Figure

As

demonstrated

in

Understanding

LGN

responses

standard

models

can

be

extended

to

include

these

adaptive

properties

and

this

extension

is

important

for

predicting

the

responses

to

complex

naturalistic

stimuli

Inclusion

of

these

dynamic

mechanisms

in

models

of

cortical

neurons

in

which

adaptation

effects

are

known

to

be

even

more

severe

than

at

earlier

stages

will

likewise

improve

their

predictive

power

Extending

these

characterization

techniques

to

include

other

known

dynamic

nonlinearities

will

also

improve

their

performance

For

example

including

a

realistic

-LRB-

non-Poisson

-RRB-

spike

generator

that

captures

a

refractory

period

and

integrate-and-fire

dynamics

significantly

improves

the

predictability

of

the

response

at

short

timescales

-LRB-

e.g.

ms

-RRB-

-LRB-

Keat

et

al.

Bair

and

Movshon

Paninski

et

al.

-RRB-

Among

all

these

limitations

a

clear

point

of

agreement

that

emerges

from

this

review

is

the

need

for

functional

models

that

provide

a

compact

description

of

the

transformation

of

stimuli

into

the

response

of

the

neuron

Functional

models

are

not

necessarily

-LRB-

and

perhaps

preferably

not

-RRB-

directly

mapped

onto

the

known

biophysics

and

anatomy

They

are

composed

of

idealized

boxes

such

as

linear

filters

divisive

stages

and

nonlinearities

all

simple

components

that

can

provide

a

compact

answer

to

the

question

What

does

this

neuron

compute

This

question

is

distinct

from

the

question

How

does

this

neuron

give

this

response

but

the

two

questions

are

clearly

related

Just

as

knowing

the

Hodgkin-Huxley

equations

has

greatly

help

the

discovery

of

how

ion

channels

work

knowing

which

computations

a

neuron

performs

on

visual

images

can

act

as

a

powerful

guide

to

understanding

the

underlying

biology

In

addition

to

guiding

the

investigation

of

underlying

biological

mechanism

a

successful

functional

model

for

a

visual

stage

is

required

if

we

want

to

understand

computation

at

later

stages

and

indeed

a

functional

model

is

what

is

needed

to

establish

the

link

between

neural

activity

and

perception

which

is

a

central

goal

of

sensory

neuroscience

References

Adelson

EH

Bergen

JR

-LRB-

-RRB-

Spatiotemporal

energy

models

for

the

perception

of

motion

J

Opt

Soc

Am

A

Albrecht

DG

Geisler

WS

-LRB-

-RRB-

Motion

selectivity

and

the

contrastresponse

function

of

simple

cells

in

the

visual

cortex

Vis

Neurosci

Albrecht

DG

Hamilton

DB

-LRB-

-RRB-

Striate

cortex

of

monkey

and

cat

contrast

response

function

J

Neurophysiol

Albright

TD

Stoner

GR

-LRB-

-RRB-

Contextual

influences

on

visual

processing

Annu

Rev

Neurosci

Andrews

BW

Pollen

DA

-LRB-

-RRB-

Relationship

between

spatial

frequency

selectivity

and

receptive

field

profile

of

simple

cells

J

Physiol

-LRB-

Lond

-RRB-

Atick

JJ

Redlich

AN

-LRB-

-RRB-

Towards

a

theory

of

early

visual

processing

Neural

Comput

Attwell

D

Laughlin

SB

-LRB-

-RRB-

An

energy

budget

for

signaling

in

the

grey

matter

of

the

brain

J

Cereb

Blood

Flow

Metab

Baccus

SA

Meister

M

-LRB-

-RRB-

Fast

and

slow

contrast

adaptation

in

retinal

circuitry

Neuron

Bair

W

Movshon

JA

-LRB-

-RRB-

Adaptive

temporal

integration

of

motion

in

direction-selective

neurons

in

macaque

visual

cortex

J

Neurosci

Baylor

DA

Hodgkin

AL

Lamb

TD

-LRB-

-RRB-

Reconstruction

of

the

electrical

responses

of

turtle

cones

to

flashes

and

steps

of

light

J

Physiol

-LRB-

Lond

-RRB-

Benardete

EA

Kaplan

E

-LRB-

-RRB-

The

dynamics

of

primate

M

retinal

ganglion

cells

Vis

Neurosci

Berry

MJ

Warland

DK

Meister

M

-LRB-

-RRB-

The

structure

and

precision

of

retinal

spike

trains

Proc

Natl

Acad

Sci

USA

Berson

DM

-LRB-

-RRB-

Strange

vision

ganglion

cells

as

circadian

photoreceptors

Trends

Neurosci

Betsch

BY

Einhauser

W

Kording

KP

Konig

P

-LRB-

-RRB-

The

world

from

a

cat

's

perspective

statistics

of

natural

videos

Biol

Cybern

Blakemore

C

Tobin

EA

-LRB-

-RRB-

Lateral

inhibition

between

orientation

detectors

in

the

cat

's

visual

cortex

Exp

Brain

Res

Bonds

AB

-LRB-

-RRB-

Role

of

inhibition

in

the

specification

of

orientation

selectivity

of

cells

in

the

cat

striate

cortex

Vis

Neurosci

Bonin

V

Mante

V

Carandini

M

-LRB-

-RRB-

The

suppressive

field

of

neurons

in

lateral

geniculate

nucleus

J

Neurosci

in

press

Brenner

N

Bialek

W

de

Ruyter

van

Steveninck

R

-LRB-

-RRB-

Adaptive

rescaling

maximizes

information

transmission

Neuron

Brodie

SE

Knight

BW

Ratliff

F

-LRB-

-RRB-

The

spatiotemporal

transfer

function

of

the

Limulus

lateral

eye

J

Gen

Physiol

Cai

D

DeAngelis

GC

Freeman

RD

-LRB-

-RRB-

Spatiotemporal

receptive

field

organization

in

the

lateral

geniculate

nucleus

of

cats

and

kittens

J

Neurophysiol

Carandini

M

-LRB-

-RRB-

Receptive

fields

and

suppressive

fields

in

the

early

visual

system

In

The

cognitive

neurosciences

III

Ed

-LRB-

Gazzaniga

MS

ed

-RRB-

pp

Cambridge

MA

MIT

Carandini

M

Ferster

D

-LRB-

-RRB-

Membrane

potential

and

firing

rate

in

cat

primary

visual

cortex

J

Neurosci

Carandini

M

Heeger

DJ

-LRB-

-RRB-

Summation

and

division

by

neurons

in

primate

visual

cortex

Science

Carandini

M

Heeger

DJ

Movshon

JA

-LRB-

-RRB-

Linearity

and

normalization

in

simple

cells

of

the

macaque

primary

visual

cortex

J

Neurosci

Carandini

M

Heeger

DJ

Movshon

JA

-LRB-

-RRB-

Linearity

and

gain

control

in

simple

cells

In

Cerebral

cortex

Vol

Models

of

cortical

circuits

-LRB-

Ulinski

PS

Jones

EG

Peters

A

eds

-RRB-

pp

New

York

Kluwer

Academic/Plenum

Carandini

M

Heeger

DJ

Senn

W

-LRB-

-RRB-

A

synaptic

explanation

of

suppression

in

visual

cortex

J

Neurosci

Chander

D

Chichilnisky

EJ

-LRB-

-RRB-

Adaptation

to

temporal

contrast

in

primate

and

salamander

retina

J

Neurosci

Cheng

H

Chino

YM

Smith

III

EL

Hamamoto

J

Yoshida

K

-LRB-

-RRB-

Transfer

J.

Neurosci.

November

-LRB-

-RRB-

characteristics

of

lateral

geniculate

nucleus

X

neurons

in

the

cat

effects

of

spatial

frequency

and

contrast

J

Neurophysiol

Chichilnisky

EJ

-LRB-

-RRB-

A

simple

white

noise

analysis

of

neuronal

light

responses

Network

Chisum

HJ

Fitzpatrick

D

-LRB-

-RRB-

The

contribution

of

veritical

and

horizontal

connections

to

the

receptive

field

center

and

surround

in

Neural

Netw

Cleland

BG

Enroth-Cugell

C

-LRB-

-RRB-

Quantitative

aspects

of

sensitivity

and

summation

in

the

cat

retina

J

Physiol

-LRB-

Lond

-RRB-

Cleland

BG

Freeman

AW

-LRB-

-RRB-

Visual

adaptation

is

highly

localized

in

the

cat

's

retina

J

Physiol

-LRB-

Lond

-RRB-

Connor

CE

Preddie

DC

Gallant

JL

Van

Essen

DC

-LRB-

-RRB-

Spatial

attention

effects

in

macaque

area

J

Neurosci

Dacey

DM

Liao

HW

Peterson

BB

Robinson

FR

Smith

VC

Pokorny

J

Yau

KW

Gamlin

PD

-LRB-

-RRB-

Melanopsin-expressing

ganglion

cells

in

primate

retina

signal

colour

and

irradiance

and

project

to

the

LGN

Nature

Dan

Y

Atick

JJ

Reid

RC

-LRB-

-RRB-

Efficient

coding

of

natural

scenes

in

the

lateral

geniculate

nucleus

experimental

test

of

a

computational

theory

J

Neurosci

Daugman

JG

-LRB-

-RRB-

Two-dimensional

spectral

analysis

of

cortical

receptive

field

profiles

Vision

Res

David

SV

Gallant

JL

-LRB-

-RRB-

Predicting

neuronal

responses

during

natural

vision

Network

in

press

David

SV

Mazer

JA

Gallant

JL

-LRB-

-RRB-

Pattern-specific

attentional

modulation

of

spatiotemporal

receptive

fields

during

free-viewing

visual

search

Soc

Neurosci

Abstr

David

SV

Vinje

WE

Gallant

JL

-LRB-

-RRB-

Natural

stimulus

statistics

alter

the

receptive

field

structure

of

neurons

J

Neurosci

Dawis

S

Shapley

R

Kaplan

E

Tranchina

D

-LRB-

-RRB-

The

receptive

field

organization

of

X-cells

in

the

cat

spatiotemporal

coupling

and

asymmetry

Vision

Res

De

Ruyter

Van

Steveninck

R

Bialek

W

-LRB-

-RRB-

Real-time

performance

of

a

movement-sensitive

neuron

in

the

blowfly

visual

system

coding

and

information

transfer

in

short

spike

sequences

Proc

R

Soc

Lond

Ser

B

Biol

Sci

De

Valois

RL

De

Valois

K

-LRB-

-RRB-

Spatial

vision

Oxford

Oxford

UP

De

Valois

RL

Albrecht

DG

Thorell

LG

-LRB-

-RRB-

Spatial

frequency

selectivity

of

cells

in

macaque

visual

cortex

Vision

Res

DeAngelis

GC

Robson

JG

Ohzawa

I

Freeman

RD

-LRB-

-RRB-

Organization

of

suppression

in

receptive

fields

of

neurons

in

cat

visual

cortex

J

Neurophysiol

DeAngelis

GC

Ohzawa

I

Freeman

RD

-LRB-

-RRB-

Spatiotemporal

organization

of

simple-cell

receptive

fields

in

the

cat

's

striate

cortex

I.

General

characteristics

and

postnatal

development

J

Neurophysiol

DeAngelis

GC

Ohzawa

I

Freeman

RD

-LRB-

-RRB-

Spatiotemporal

organization

of

simple-cell

receptive

fields

in

the

cat

's

striate

cortex

II

Linearity

of

temporal

and

spatial

summation

J

Neurophysiol

Dean

AF

Tolhurst

DJ

-LRB-

-RRB-

On

the

distinctness

of

simple

and

complex

cells

in

the

visual

cortex

of

the

cat

J

Physiol

-LRB-

Lond

-RRB-

Dean

AF

Tolhurst

DJ

-LRB-

-RRB-

Factors

influencing

the

temporal

phase

of

response

to

bar

and

grating

stimuli

for

simple

cells

in

the

cat

striate

cortex

Exp

Brain

Res

DeBoer

E

Kuyper

P

-LRB-

-RRB-

Triggered

correlation

IEEE

Trans

Biomed

Eng

Demb

JB

-LRB-

-RRB-

Multiple

mechanisms

for

contrast

adaptation

in

the

retina

Neuron

Demb

JB

Haarsma

L

Freed

MA

Sterling

P

-LRB-

-RRB-

Functional

circuitry

of

the

retinal

ganglion

cell

's

nonlinear

receptive

field

J

Neurosci

Demb

JB

Zaghloul

K

Haarsma

L

Sterling

P

-LRB-

-RRB-

Bipolar

cells

contribute

to

nonlinear

spatial

summation

in

the

brisk-transient

-LRB-

Y

-RRB-

ganglion

cell

in

mammalian

retina

J

Neurosci

Demb

JB

Sterling

P

Freed

MA

-LRB-

-RRB-

How

retinal

ganglion

cells

prevent

synaptic

noise

from

reaching

the

spike

output

J

Neurophysiol

Derrington

AM

Lennie

P

-LRB-

-RRB-

Spatial

and

temporal

contrast

sensitivities

of

neurons

in

lateral

geniculate

nucleus

of

macaque

J

Physiol

-LRB-

Lond

-RRB-

Desimone

R

Schein

SJ

-LRB-

-RRB-

Visual

properties

of

neurons

in

area

of

the

macaque

sensitivity

to

stimulus

form

J

Neurophysiol

Dong

DW

Atick

JJ

-LRB-

-RRB-

Temporal

decorrelation

a

theory

of

lagged

and

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

nonlagged

responses

in

the

lateral

geniculate

nucleus

Network

Edin

F

Machens

CK

Schutze

H

Herz

AV

-LRB-

-RRB-

Searching

for

optimal

sensory

signals

iterative

stimulus

reconstruction

in

closed-loop

experiments

J

Comput

Neurosci

Eggermont

JJ

-LRB-

-RRB-

Wiener

and

Volterra

analyses

applied

to

the

auditory

system

Hear

Res

Emerson

RC

Citron

MC

Vaughn

WJ

Klein

SA

-LRB-

-RRB-

Nonlinear

directionally

selective

subunits

in

complex

cells

of

cat

striate

cortex

J

Neurophysiol

Enroth-Cugell

C

Freeman

AW

-LRB-

-RRB-

The

receptive-field

spatial

structure

of

cat

retinal

Y

cells

J

Physiol

-LRB-

Lond

-RRB-

Enroth-Cugell

C

Jakiela

HG

-LRB-

-RRB-

Suppression

of

cat

retinal

ganglion

cell

responses

by

moving

patterns

J

Physiol

-LRB-

Lond

-RRB-

Enroth-Cugell

C

Robson

JG

-LRB-

-RRB-

The

contrast

sensitivity

of

retinal

ganglion

cells

of

the

cat

J

Physiol

-LRB-

Lond

-RRB-

Enroth-Cugell

C

Shapley

RM

-LRB-

-RRB-

Adaptation

and

dynamics

of

cat

retinal

ganglion

cells

J

Physiol

-LRB-

Lond

-RRB-

Enroth-Cugell

C

Shapley

RM

-LRB-

-RRB-

Flux

not

retinal

illumination

is

what

cat

retinal

ganglion

cells

really

care

about

J

Physiol

-LRB-

Lond

-RRB-

Enroth-Cugell

C

Lennie

P

Shapley

RM

-LRB-

-RRB-

Surround

contribution

to

light

adaptation

in

cat

retinal

ganglion

cells

J

Physiol

-LRB-

Lond

-RRB-

Felsen

G

Touryan

J

Han

F

Dan

Y

-LRB-

-RRB-

Cortical

sensitivity

to

visual

features

in

natural

scenes

PLoS

Biol

Ferster

D

-LRB-

-RRB-

Spatially

opponent

excitation

and

inhibition

in

simple

cells

of

the

cat

visual

cortex

J

Neurosci

Field

DJ

-LRB-

-RRB-

Relations

between

the

statistics

of

natural

images

and

the

response

properties

of

cortical

cells

J

Opt

Soc

Am

A

Field

DJ

Tolhurst

DJ

-LRB-

-RRB-

The

structure

and

symmetry

of

simple-cell

receptive-field

profiles

in

the

cat

's

visual

cortex

Proc

R

Soc

Lond

B

Biol

Sci

Fitzpatrick

D

-LRB-

-RRB-

Seeing

beyond

the

receptive

field

in

primary

visual

cortex

Curr

Opin

Neurobiol

Foldiak

P

Xiao

D

Keysers

C

Edwards

R

Perrett

DI

-LRB-

-RRB-

Rapid

serial

visual

presentation

for

the

determination

of

neural

selectivity

in

area

STSa

Prog

Brain

Res

Freeman

RD

Ohzawa

I

Walker

G

-LRB-

-RRB-

Beyond

the

classical

receptive

field

in

the

visual

cortex

Prog

Brain

Res

Freeman

T

Durand

S

Kiper

D

Carandini

M

-LRB-

-RRB-

Suppression

without

inhibition

in

visual

cortex

Neuron

Fuortes

MGF

Hodgkin

AL

-LRB-

-RRB-

Changes

in

the

time

scale

and

sensitiviy

in

the

ommatidia

of

Limulus

J

Physiol

-LRB-

Lond

-RRB-

Gallant

JL

-LRB-

-RRB-

Neural

mechanisms

of

natural

scene

perception

In

The

visual

neurosciences

-LRB-

Chalupa

LM

Werner

JS

eds

-RRB-

Boston

MIT

Gallant

JL

Connor

CE

Rakshit

S

Lewis

JW

Van

Essen

DC

-LRB-

-RRB-

Neural

responses

to

polar

hyperbolic

and

Cartesian

gratings

in

area

of

the

macaque

monkey

J

Neurophysiol

Gardner

JL

Anzai

A

Ohzawa

I

Freeman

RD

-LRB-

-RRB-

Linear

and

nonlinear

contributions

to

orientation

tuning

of

simple

cells

in

the

cat

's

striate

cortex

Vis

Neurosci

Glezer

VD

Tsherbach

TA

Gauselman

VE

Bondarko

VM

-LRB-

-RRB-

Spatiotemporal

organization

of

receptive

fields

of

the

cat

striate

cortex

The

receptive

fields

as

the

grating

filters

Biol

Cybern

Graham

NVS

-LRB-

-RRB-

Visual

pattern

analyzers

New

York

Oxford

UP

Guido

W

Weyand

T

-LRB-

-RRB-

Burst

responses

in

thalamic

relay

cells

of

the

awake

behaving

cat

J

Neurophysiol

Guillery

RW

Sherman

SM

-LRB-

-RRB-

Thalamic

relay

functions

and

their

role

in

corticocortical

communication

generalizations

from

the

visual

system

Neuron

Hayden

BY

Gallant

JL

-LRB-

-RRB-

Timecourse

of

attention

reveals

different

mechanisms

for

spatial

and

feature

attention

in

Neuron

Heeger

DJ

-LRB-

-RRB-

Normalization

of

cell

responses

in

cat

striate

cortex

Vis

Neurosci

Heeger

DJ

-LRB-

-RRB-

Half-squaring

in

responses

of

cat

striate

cells

Vis

Neurosci

Heeger

DJ

-LRB-

-RRB-

Modeling

simple-cell

direction

selectivity

with

normalized

half-squared

linear

operators

J

Neurophysiol

Heeger

DJ

Bergen

JR

-LRB-

-RRB-

Pyramid

based

texture

analysis/synthesis

In

Proceedings

of

the

annual

conference

on

computer

graphics

and

interactive

techniques

pp

New

York

ACM

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

Hirsch

JA

-LRB-

-RRB-

Synaptic

physiology

and

receptive

field

structure

in

the

early

visual

pathway

of

the

cat

Cereb

Cortex

Hirsch

JA

Alonso

JM

Reid

RC

Martinez

LM

-LRB-

-RRB-

Synaptic

integration

in

striate

cortical

simple

cells

J

Neurosci

Hochstein

S

Shapley

RM

-LRB-

-RRB-

Linear

and

nonlinear

spatial

subunits

in

Y

cat

retinal

ganglion

cells

J

Physiol

-LRB-

Lond

-RRB-

Hosoya

T

Baccus

SA

Meister

M

-LRB-

-RRB-

Dynamic

predictive

coding

by

the

retina

Nature

Hsu

A

Borst

A

Theunissen

FE

-LRB-

-RRB-

Quantifying

variability

in

neural

responses

and

its

application

for

the

validation

of

model

predictions

Network

Hubel

DH

Wiesel

TN

-LRB-

-RRB-

Receptive

fields

of

single

neurones

in

the

cat

's

striate

cortex

J

Physiol

-LRB-

Lond

-RRB-

Hubel

DH

Wiesel

TN

-LRB-

-RRB-

Receptive

fields

binocular

interaction

and

functional

architecture

in

the

cat

's

visual

cortex

J

Physiol

-LRB-

Lond

-RRB-

Jagadeesh

B

Wheat

HS

Ferster

D

-LRB-

-RRB-

Linearity

of

summation

of

synaptic

potentials

underlying

direction

selectivity

in

simple

cells

of

the

cat

visual

cortex

Science

Jones

HE

Andolina

IM

Oakely

NM

Murphy

PC

Sillito

AM

-LRB-

-RRB-

Spatial

summation

in

lateral

geniculate

nucleus

and

visual

cortex

Exp

Brain

Res

Jones

JP

Palmer

LA

-LRB-

-RRB-

The

two-dimensional

spatial

structure

of

simple

receptive

fields

in

cat

striate

cortex

J

Neurophysiol

Kaplan

E

Marcus

S

So

YT

-LRB-

-RRB-

Effects

of

dark

adaptation

on

spatial

and

temporal

properties

of

receptive

fields

in

cat

lateral

geniculate

nucleus

J

Physiol

-LRB-

Lond

-RRB-

Kaplan

E

Purpura

K

Shapley

R

-LRB-

-RRB-

Contrast

affects

the

transmission

of

visual

information

through

the

mammalian

lateral

geniculate

nucleus

J

Physiol

-LRB-

Lond

-RRB-

Kara

P

Reinagel

P

Reid

RC

-LRB-

-RRB-

Low

response

variability

in

simultaneously

recorded

retinal

thalamic

and

cortical

neurons

Neuron

Kayser

C

Salazar

receptive

field

Konig

P

-LRB-

-RRB-

Responses

to

natural

scenes

in

cat

J

Neurophysiol

Keat

J

Reinagel

P

Reid

RC

Meister

M

-LRB-

-RRB-

Predicting

every

spike

a

model

for

the

responses

of

visual

neurons

Neuron

Kim

KJ

Rieke

F

-LRB-

-RRB-

Temporal

contrast

adaptation

in

the

input

and

output

signals

of

salamander

retinal

ganglion

cells

J

Neurosci

Kim

KJ

Rieke

F

-LRB-

-RRB-

Slow

inactivation

and

variance

adaptation

in

salamander

retinal

ganglion

cells

J

Neurosci

Knierim

JJ

van

Essen

DC

-LRB-

-RRB-

Neuronal

responses

to

static

texture

patterns

in

area

of

the

alert

macaque

monkey

J

Neurophysiol

Kremers

J

Weiss

S

Zrenner

E

-LRB-

-RRB-

Temporal

properties

of

marmoset

lateral

geniculate

cells

Vision

Res

Kuffler

SW

-LRB-

-RRB-

Discharge

patterns

and

functional

organization

of

mammalian

retina

J

Neurophysiol

Kuffler

SW

Fitzhugh

R

Barlow

HB

-LRB-

-RRB-

Maintained

activity

in

the

cat

's

retina

in

light

and

darkness

J

Gen

Physiol

Lampl

I

Anderson

JS

Gillespie

DC

Ferster

D

-LRB-

-RRB-

Prediction

of

orientation

selectivity

from

receptive

field

architecture

in

simple

cells

of

cat

visual

cortex

Neuron

Lankheet

MJ

Van

Wezel

RJ

Prickaerts

JH

van

de

Grind

WA

-LRB-

-RRB-

The

dynamics

of

light

adaptation

in

cat

horizontal

cell

responses

Vision

Res

Lankheet

MJ

Przybyszewski

AW

van

de

Grind

WA

-LRB-

-RRB-

The

lateral

spread

of

light

adaptation

in

cat

horizontal

cell

responses

Vision

Res

Lau

B

Stanley

GB

Dan

Y

-LRB-

-RRB-

Computational

subunits

of

visual

cortical

neurons

revealed

by

artificial

neural

networks

Proc

Natl

Acad

Sci

USA

Lee

BB

Dacey

DM

Smith

VC

Pokorny

J

-LRB-

-RRB-

Dynamics

of

sensitivity

regulation

in

primate

outer

retina

the

horizontal

cell

network

J

Vis

Lee

TS

Mumford

D

-LRB-

-RRB-

Hierarchical

Bayesian

inference

in

the

visual

cortex

J

Opt

Soc

Am

A

Opt

Image

Sci

Vis

Lehky

SR

Sejnowski

TJ

Desimone

R

-LRB-

-RRB-

Predicting

responses

of

nonlinear

neurons

in

monkey

striate

cortex

to

complex

patterns

J

Neurosci

Lennie

P

-LRB-

-RRB-

The

cost

of

cortical

computation

Curr

Biol

Lesica

Stanley

GB

-LRB-

-RRB-

Encoding

of

natural

scene

movies

by

tonic

J.

Neurosci.

November

-LRB-

-RRB-

and

burst

spikes

in

the

lateral

geniculate

nucleus

J

Neurosci

Lesica

Boloori

AS

Stanley

GB

-LRB-

-RRB-

Adaptive

encoding

in

the

visual

pathway

Network

Li

B

Peterson

MR

Thompson

JK

Duong

T

Freeman

RD

-LRB-

-RRB-

Crossorientation

suppression

monoptic

and

dichoptic

mechanisms

are

different

J

Neurophysiol

Maffei

L

Fiorentini

A

-LRB-

-RRB-

The

unresponsive

regions

of

visual

cortical

receptive

fields

Vision

Res

Maffei

L

Fiorentini

A

Bisti

S

-LRB-

-RRB-

Neural

correlate

of

perceptual

adaptation

to

gratings

Science

Mante

V

-LRB-

-RRB-

Gain

controls

based

on

luminance

and

contrast

in

the

early

visual

system

In

Institute

of

neuroinformatics

PhD

thesis

Federal

Institute

of

Technology

Zurich

Mante

V

Bonin

V

Carandini

M

-LRB-

-RRB-

How

local

contrast

determines

gain

and

temporal

integration

in

LGN

Soc

Neurosci

Abstr

Mante

V

Frazor

RA

Bonin

V

Geisler

WS

Carandini

M

-LRB-

-RRB-

Independence

of

luminance

and

contrast

in

natural

scenes

and

in

the

early

visual

system

Nat

Neurosci

in

press

Marmarelis

PZ

Marmarelis

VZ

-LRB-

-RRB-

Analysis

of

physiological

systems

the

white

noise

approach

New

York

Plenum

Martinez

LM

Wang

Q

Reid

RC

Pillai

C

Alonso

JM

Sommer

FT

Hirsch

JA

-LRB-

-RRB-

Receptive

field

structure

varies

with

layer

in

the

primary

visual

cortex

Nat

Neurosci

Masland

RH

-LRB-

-RRB-

Neuronal

diversity

in

the

retina

Curr

Opin

Neurobiol

Mata

ML

Ringach

DL

-LRB-

-RRB-

Spatial

overlap

of

ON

and

OFF

subregions

and

its

relation

to

response

modulation

ratio

in

macaque

primary

visual

cortex

J

Neurophysiol

Mazer

JA

Gallant

JL

-LRB-

-RRB-

Goal-related

activity

in

during

free

viewing

visual

search

Evidence

for

a

ventral

stream

visual

salience

map

Neuron

McLean

J

Palmer

LA

-LRB-

-RRB-

Contribution

of

linear

spatiotemporal

receptive

field

structure

to

velocity

selectivity

of

simple

cells

in

area

of

cat

Vision

Res

Mechler

F

Ringach

DL

-LRB-

-RRB-

On

the

classification

of

simple

and

complex

cells

Vision

Res

Movshon

JA

Thompson

ID

Tolhurst

DJ

-LRB-

-RRB-

Spatial

summation

in

the

receptive

fields

of

simple

cells

in

the

cat

's

striate

cortex

J

Physiol

-LRB-

Lond

-RRB-

Movshon

JA

Thompson

ID

Tolhurst

DJ

-LRB-

-RRB-

Receptive

field

organization

of

complex

cells

in

the

cat

's

striate

cortex

J

Physiol

-LRB-

Lond

-RRB-

Mukherjee

P

Kaplan

E

-LRB-

-RRB-

Dynamics

of

neurons

in

the

cat

lateral

geniculate

nucleus

in

vivo

electrophysiology

and

computational

modeling

J

Neurophysiol

Nakayama

K

He

ZJ

Shimojo

S

-LRB-

-RRB-

Visual

surface

representation

a

critical

link

between

lower-level

and

higher-level

vision

In

An

invitation

to

cognitive

science

-LRB-

Kosslyn

SM

Osherson

DN

eds

-RRB-

pp

Cambridge

MA

MIT

O'Connor

KN

Petkov

CI

Sutter

ML

-LRB-

-RRB-

Stimulus

optimization

for

auditory

cortical

neurons

Soc

Neurosci

Abstr

Olshausen

BA

Field

DJ

-LRB-

-RRB-

Sparse

coding

of

sensory

inputs

Curr

Opin

Neurobiol

Olshausen

BA

Field

DJ

-LRB-

-RRB-

How

close

are

we

to

understanding

Neural

Comput

Olveczky

BP

Baccus

SA

Meister

M

-LRB-

-RRB-

Segregation

of

object

and

background

motion

in

the

retina

Nature

Ozeki

H

Sadakane

O

Akasaki

T

Naito

T

Shimegi

S

Sato

H

-LRB-

-RRB-

Relationship

between

excitation

and

inhibition

underlying

size

tuning

and

contextual

response

modulation

in

the

cat

primary

visual

cortex

J

Neurosci

Palmer

LA

Davis

TL

-LRB-

-RRB-

Receptive-field

structure

in

cat

striate

cortex

J

Neurophysiol

Paninski

L

Pillow

JW

Simoncelli

EP

-LRB-

-RRB-

Maximum

likelihood

estimation

of

a

stochastic

integrate-and-fire

neural

encoding

model

Neural

Comput

Passaglia

CL

Enroth-Cugell

C

Troy

JB

-LRB-

-RRB-

Effects

of

remote

stimulation

on

the

mean

firing

rate

of

cat

retinal

ganglion

cells

J

Neurosci

Pillow

JW

Simoncelli

EP

-LRB-

-RRB-

Biases

in

white

noise

analysis

due

to

nonPoisson

spike

generation

Neurocomputing

J.

Neurosci.

November

-LRB-

-RRB-

Prenger

R

Wu

MC

David

SV

Gallant

JL

-LRB-

-RRB-

Nonlinear

responses

to

natural

scenes

revealed

by

neural

network

analysis

Neural

Netw

Priebe

NJ

Mechler

F

Carandini

M

Ferster

D

-LRB-

-RRB-

The

contribution

of

spike

threshold

to

the

dichotomy

of

cortical

simple

and

complex

cells

Nat

Neurosci

Purpura

K

Tranchina

D

Kaplan

E

Shapley

RM

-LRB-

-RRB-

Light

adaptation

in

the

primate

retina

analysis

of

changes

in

gain

and

dynamics

of

monkey

retinal

ganglion

cells

Vis

Neurosci

Ramcharan

EJ

Gnadt

JW

Sherman

SM

-LRB-

-RRB-

Higher-order

thalamic

relays

burst

more

than

first-order

relays

Proc

Natl

Acad

Sci

USA

Reich

DS

Victor

JD

Knight

BW

Ozaki

T

Kaplan

E

-LRB-

-RRB-

Response

variability

and

timing

precision

of

neuronal

spike

trains

in

vivo

J

Neurophysiol

Reid

RC

Soodak

RE

Shapley

RM

-LRB-

-RRB-

Directional

selectivity

and

spatiotemporal

structure

of

receptive

fields

of

simple

cells

in

cat

striate

cortex

J

Neurophysiol

Reid

RC

Victor

JD

Shapley

RM

-LRB-

-RRB-

The

use

of

m-sequences

in

the

analysis

of

visual

neurons

linear

receptive

field

properties

Vis

Neurosci

Rieke

F

-LRB-

-RRB-

Temporal

contrast

adaptation

in

salamander

bipolar

cells

J

Neurosci

Ringach

DL

Hawken

MJ

Shapley

R

-LRB-

-RRB-

Receptive

field

structure

of

neurons

in

monkey

primary

visual

cortex

revealed

by

stimulation

with

natural

image

sequences

J

Vis

Robson

JG

-LRB-

-RRB-

Receptive

fields

spatial

and

intensive

representation

of

the

visual

image

In

Handbook

of

perception

-LRB-

Carterette

D

Friedman

W

eds

-RRB-

New

York

Academic

Rodieck

RW

-LRB-

-RRB-

The

first

steps

in

seeing

Sunderland

MA

Sinauer

Rodieck

RW

Brening

RK

Watanabe

M

-LRB-

-RRB-

The

origin

of

parallel

visual

pathways

In

Contrast

sensitivity

-LRB-

Shapley

RM

Man-Kit

Lam

D

eds

-RRB-

Cambridge

MA

Bradford

Books

Roska

B

Werblin

F

-LRB-

-RRB-

Vertical

interactions

across

ten

parallel

stacked

representations

in

the

mammalian

retina

Nature

Rust

NC

Schwartz

O

Movshon

JA

Simoncelli

EP

-LRB-

-RRB-

Spatiotemporal

elements

of

macaque

receptive

fields

Neuron

Sahani

M

Linden

JF

-LRB-

-RRB-

How

linear

are

auditory

cortical

responses

In

Advances

in

neural

information

processing

systems

-LRB-

Becker

S

Thrun

S

Obermeyer

K

eds

-RRB-

pp

Cambridge

MA

MIT

Saito

H

Fukada

Y

-LRB-

-RRB-

Gain

control

mechanisms

in

X

and

Y-type

retinal

ganglion

cells

of

the

cat

Vision

Res

Sakai

HM

Naka

K

-LRB-

-RRB-

Response

dynamics

and

receptive-field

organization

of

catfish

ganglion

cells

J

Gen

Physiol

Saul

AB

Humphrey

AL

-LRB-

-RRB-

Spatial

and

temporal

response

properties

of

lagged

and

nonlagged

cells

in

cat

lateral

geniculate

nucleus

J

Neurophysiol

Schumer

RA

Movshon

JA

-LRB-

-RRB-

Length

summation

in

simple

cells

of

cat

striate

cortex

Vision

Res

Sclar

G

-LRB-

-RRB-

Expression

of

retinal

contrast

gain

control

by

neurons

of

the

cat

's

lateral

geniculate

nucleus

Exp

Brain

Res

Sclar

G

Maunsell

JHR

Lennie

P

-LRB-

-RRB-

Coding

of

image

contrast

in

central

visual

pathways

of

the

macaque

monkey

Vision

Res

Sengpiel

F

Vorobyov

V

-LRB-

-RRB-

Intracortical

origins

of

interocular

suppression

in

the

visual

cortex

J

Neurosci

Sengpiel

F

Baddeley

RJ

Freeman

TC

Harrad

R

Blakemore

C

-LRB-

-RRB-

Different

mechanisms

underlie

three

inhibitory

phenomena

in

cat

area

Vision

Res

Series

P

Lorenceau

J

Fregnac

Y

-LRB-

-RRB-

The

silent

surround

of

receptive

fields

theory

and

experiments

J

Physiol

-LRB-

Paris

-RRB-

Shapley

R

Lennie

P

-LRB-

-RRB-

Spatial

frequency

analysis

in

the

visual

system

Annu

Rev

Neurosci

Shapley

R

Victor

JD

-LRB-

-RRB-

Nonlinear

spatial

summation

and

the

contrast

gain

control

of

cat

retinal

ganglion

cells

J

Physiol

-LRB-

Lond

-RRB-

Shapley

RM

Enroth-Cugell

C

-LRB-

-RRB-

Visual

adaptation

and

retinal

gain

controls

In

Progress

in

retinal

research

-LRB-

Osborne

N

Chader

G

eds

-RRB-

pp

London

Pergamon

Shapley

RM

Victor

JD

-LRB-

-RRB-

The

effect

of

contrast

on

the

transfer

properties

of

cat

retinal

ganglion

cells

J

Physiol

-LRB-

Lond

-RRB-

Shapley

RM

Victor

JD

-LRB-

-RRB-

How

the

contrast

gain

control

modifies

the

frequency

responses

of

cat

retinal

ganglion

cells

J

Physiol

-LRB-

Lond

-RRB-

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

Sharpee

T

Rust

NC

Bialek

W

-LRB-

-RRB-

Analyzing

neural

responses

to

natural

signals

maximally

informative

dimensions

Neural

Comput

Sherman

SM

-LRB-

-RRB-

Tonic

and

burst

firing

dual

modes

of

thalamocortical

relay

Trends

Neurosci

Simoncelli

EP

Heeger

DJ

-LRB-

-RRB-

A

model

of

neuronal

responses

in

visual

area

MT.

Vision

Res

Simoncelli

EP

Olshausen

BA

-LRB-

-RRB-

Natural

image

statistics

and

neural

representation

Annu

Rev

Neurosci

Simoncelli

EP

Pillow

JW

Paninski

L

Schwartz

O

-LRB-

-RRB-

Characterization

of

neural

responses

with

stochastic

stimuli

In

The

cognitive

neurosciences

-LRB-

Gazzaniga

M

ed

-RRB-

pp

Cambridge

MA

MIT

Smirnakis

SM

Berry

MJ

Warland

DK

Bialek

W

Meister

M

-LRB-

-RRB-

Adaptation

of

retinal

processing

to

image

contrast

and

spatial

scale

Nature

Smith

GD

Cox

CL

Sherman

SM

Rinzel

J

-LRB-

-RRB-

Fourier

analysis

of

sinusoidally

driven

thalamocortical

relay

neurons

and

a

minimal

integrateand-fire-or-burst

model

J

Neurophysiol

Smyth

D

Willmore

B

Baker

GE

Thompson

ID

Tolhurst

DJ

-LRB-

-RRB-

The

receptive-field

organization

of

simple

cells

in

primary

visual

cortex

of

ferrets

under

natural

scene

stimulation

J

Neurosci

So

YT

Shapley

R

-LRB-

-RRB-

Spatial

tuning

of

cells

in

and

around

lateral

geniculate

nucleus

of

the

cat

X

and

Y

relay

cells

and

perigeniculate

interneurons

J

Neurophysiol

Solomon

SG

White

AJ

Martin

PR

-LRB-

-RRB-

Extraclassical

receptive

field

properties

of

parvocellular

magnocellular

and

koniocellular

cells

in

the

primate

lateral

geniculate

nucleus

J

Neurosci

Sterling

P

Demb

JB

-LRB-

-RRB-

Retina

In

Synaptic

organization

of

the

brain

Ed

-LRB-

Sheppard

G

ed

-RRB-

pp

New

York

Oxford

UP

Sutter

EE

-LRB-

-RRB-

A

revised

conception

of

visual

receptive

fields

based

on

pseudorandom

spatiotemporal

pattern

stimuli

In

Proceedings

of

the

first

symposium

on

testing

and

identification

of

nonlinear

systems

-LRB-

Mc

Cann

GD

Marmarelis

PZ

eds

-RRB-

pp

Pasadena

CA

California

Institute

of

Technology

Tadmor

Y

Tolhurst

DJ

-LRB-

-RRB-

The

effect

of

threshold

on

the

relationship

between

the

receptive-field

profile

and

the

spatial-frequency

tuning

curve

in

simple

cells

of

the

cat

's

striate

cortex

Vis

Neurosci

Taylor

WR

Vaney

DI

-LRB-

-RRB-

New

directions

in

retinal

research

Trends

Neurosci

Theunissen

FE

Sen

K

Doupe

AJ

-LRB-

-RRB-

Spectral-temporal

receptive

fields

of

nonlinear

auditory

neurons

obtained

using

natural

sounds

J

Neurosci

Theunissen

FE

David

SV

Singh

NC

Hsu

A

Vinje

WE

Gallant

JL

-LRB-

-RRB-

Estimating

spatio-temporal

receptive

fields

of

auditory

and

visual

neurons

from

their

responses

to

natural

stimuli

Network

Tolhurst

DJ

Dean

AF

-LRB-

-RRB-

Spatial

summation

by

simple

cells

in

the

striate

cortex

of

the

cat

Exp

Brain

Res

Tolhurst

DJ

Dean

AF

-LRB-

-RRB-

The

effects

of

contrast

on

the

linearity

of

spatial

summation

of

simple

cells

in

the

cat

's

striate

cortex

Exp

Brain

Res

Tolhurst

DJ

Dean

AF

-LRB-

-RRB-

Evaluation

of

a

linear

model

of

directional

selectivity

in

simple

cells

of

the

cat

's

striate

cortex

Vis

Neurosci

Tolhurst

DJ

Heeger

DJ

-LRB-

-RRB-

Contrast

normalization

and

a

linear

model

for

the

directional

selectivity

of

simple

cells

in

cat

striate

cortex

Vis

Neurosci

Tolhurst

DJ

Heeger

DJ

-LRB-

-RRB-

Comparison

of

contrast-normalization

and

threshold

models

of

the

responses

of

simple

cells

in

cat

striate

cortex

Vis

Neurosci

Touryan

J

Lau

B

Dan

Y

-LRB-

-RRB-

Isolation

of

relevant

visual

features

from

random

stimuli

for

cortical

complex

cells

J

Neurosci

Touryan

J

Felsen

G

Dan

Y

-LRB-

-RRB-

Spatial

structure

of

complex

cell

receptive

fields

measured

with

natural

images

Neuron

Troy

JB

Enroth-Cugell

C

-LRB-

-RRB-

X

and

Y

ganglion

cells

inform

the

cat

's

brain

about

contrast

in

the

retinal

image

Exp

Brain

Res

Troy

JB

Robson

JG

-LRB-

-RRB-

Steady

discharges

of

X

and

Y

retinal

ganglion

cells

of

cat

under

photopic

illuminance

Vis

Neurosci

Uzzell

VJ

Chichilnisky

EJ

-LRB-

-RRB-

Precision

of

spike

trains

in

primate

retinal

ganglion

cells

J

Neurophysiol

van

Hateren

JH

Ruttiger

L

Sun

H

Lee

BB

-LRB-

-RRB-

Processing

of

natural

temporal

stimuli

by

macaque

retinal

ganglion

cells

J

Neurosci

Carandini

et

al.

Do

We

Know

What

the

Early

Visual

System

Does

Victor

J

-LRB-

-RRB-

The

dynamics

of

the

cat

retinal

X

cell

centre

J

Physiol

-LRB-

Lond

-RRB-

Victor

JD

-LRB-

-RRB-

Nonlinear

systems

analysis

comparison

of

white

noise

and

sum

of

sinusoids

in

a

biological

system

Proc

Natl

Acad

Sci

USA

Victor

JD

Shapley

RM

-LRB-

-RRB-

Receptive

field

mechanisms

of

cat

X

and

Y

retinal

ganglion

cells

J

Gen

Physiol

Vinje

WE

Gallant

JL

-LRB-

-RRB-

Sparse

coding

and

decorrelation

in

primary

visual

cortex

during

natural

vision

Science

Vinje

WE

Gallant

JL

-LRB-

-RRB-

Natural

stimulation

of

the

nonclassical

receptive

field

increases

information

transmission

efficiency

in

J

Neurosci

Wassle

H

-LRB-

-RRB-

Parallel

processing

in

the

mammalian

retina

Nat

Rev

Neu

rosci

Watson

AB

-LRB-

-RRB-

Efficiency

of

a

model

human

image

code

J

Opt

Soc

Am

A

Watson

AB

Solomon

JA

-LRB-

-RRB-

Model

of

visual

contrast

gain

control

and

pattern

masking

J

Opt

Soc

Am

A

Opt

Image

Sci

Vis

Weliky

M

Fiser

J

Hunt

RH

Wagner

DN

-LRB-

-RRB-

Coding

of

natural

scenes

in

primary

visual

cortex

Neuron

J.

Neurosci.

November

-LRB-

-RRB-

Wielaard

DJ

Shelley

M

McLaughlin

D

Shapley

R

-LRB-

-RRB-

How

simple

cells

are

made

in

a

nonlinear

network

model

of

the

visual

cortex

J

Neurosci

Willmore

B

Smyth

D

-LRB-

-RRB-

Methods

for

first-order

kernel

estimation

simple-cell

receptive

fields

from

responses

to

natural

scenes

Network

Willmore

B

Tolhurst

DJ

-LRB-

-RRB-

Characterizing

the

sparseness

of

neural

codes

Network

Wu

M-C

Gallant

JL

-LRB-

-RRB-

What

's

beyond

the

second

order

Kernel

regression

techniques

for

nonlinear

functional

characterization

of

visual

neurons

Soc

Neurosci

Abstr

Yeh

T

Lee

BB

Kremers

J

-LRB-

-RRB-

The

time

course

of

adaptation

in

macaque

retinal

ganglion

cells

Vision

Res

Zaghloul

KA

Boahen

K

Demb

JB

-LRB-

-RRB-

Different

circuits

for

ON

and

OFF

retinal

ganglion

cells

cause

different

contrast

sensitivities

J

Neurosci

Zaghloul

KA

Boahen

K

Demb

JB

-LRB-

-RRB-

Contrast

adaptation

in

subthreshold

and

spiking

responses

of

mammalian

Y-type

retinal

ganglion

cells

J

Neurosci

review visual neurophysiology model vision