Genome-Wide

Mapping

of

in

Vivo

Protein-DNA

Interactions

David

S.

Johnson

et

al.

Science

-LRB-

-RRB-

DOI

science

The

following

resources

related

to

this

article

are

available

online

at

-LRB-

this

information

is

current

as

of

May

-RRB-

Updated

information

and

services

including

high-resolution

figures

can

be

found

in

the

online

version

of

this

article

at

Supporting

Online

Material

can

be

found

at

A

list

of

selected

additional

articles

on

the

Science

Web

sites

related

to

this

article

can

be

found

at

This

article

cites

articles

of

which

can

be

accessed

for

free

This

article

has

been

cited

by

article

-LRB-

s

-RRB-

on

the

ISI

Web

of

Science

This

article

has

been

cited

by

articles

hosted

by

HighWire

Press

see

This

article

appears

in

the

following

subject

collections

Molecular

Biology

Information

about

obtaining

reprints

of

this

article

or

about

obtaining

permission

to

reproduce

this

article

in

whole

or

in

part

can

be

found

at

Science

-LRB-

print

ISSN

online

ISSN

-RRB-

is

published

weekly

except

the

last

week

in

December

by

the

American

Association

for

the

Advancement

of

Science

New

York

Avenue

NW

Washington

DC

Copyright

by

the

American

Association

for

the

Advancement

of

Science

all

rights

reserved

The

title

Science

is

a

registered

trademark

of

AAAS

Downloaded

from

on

May

REPORTS

sponse

in

cutaneous

CHS

The

finding

that

topical

application

of

reduced

allergic

inflammation

points

to

the

promising

potential

of

developing

pharmacological

treatments

-LRB-

-RRB-

with

the

use

of

selective

CB

receptor

agonists

or

FAAH

inhibitors

References

and

Notes

S.

Grabbe

T.

Schwarz

Immunol

Today

-LRB-

-RRB-

T.

Bisogno

A.

Ligresti

V.

Di

Marzo

Pharmacol

Biochem

Behav

-LRB-

-RRB-

M.

M.

Ibrahim

et

al.

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

M.

Maccarrone

et

al.

J.

Biol

Chem

-LRB-

-RRB-

Materials

and

methods

are

available

as

supporting

material

on

Science

Online

A.

Zimmer

A.

M.

Zimmer

A.

G.

Hohmann

M.

Herkenham

T.

I.

Bonner

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

N.

E.

Buckley

et

al.

Eur

J.

Pharmacol

-LRB-

-RRB-

S.

Werner

et

al.

Science

-LRB-

-RRB-

J.

Knop

R.

Stremmer

C.

Neumann

E.

De

Maeyer

E.

Macher

Nature

-LRB-

-RRB-

R.

I.

Lehrer

J.

Hanifin

M.

J.

Cline

Nature

-LRB-

-RRB-

C.

Nathan

Nat

Rev.

Immunol

-LRB-

-RRB-

S.

Oka

et

al.

J.

Immunol

-LRB-

-RRB-

Y.

Ueda

N.

Miyagawa

T.

Matsui

T.

Kaya

H.

Iwamura

Eur

J.

Pharmacol

-LRB-

-RRB-

V.

Di

Marzo

et

al.

J.

Neurochem

-LRB-

-RRB-

S.

Stander

M.

Schmelz

D.

Metze

T.

Luger

R.

Rukwied

J.

Dermatol

Sci

-LRB-

-RRB-

C.

A.

Lunn

et

al.

J.

Pharmacol

Exp

Ther

-LRB-

-RRB-

B.

F.

Cravatt

et

al.

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

B.

F.

Cravatt

et

al.

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

J.

S.

Lee

G.

Katari

R.

Sachidanandam

BMC

Bioinformat

-LRB-

-RRB-

M.

Ashburner

et

al.

Nat

Genet

-LRB-

-RRB-

M.

F.

Bachmann

M.

Kopf

B.

J.

Marsland

Nat

Rev.

Immunol

-LRB-

-RRB-

A.

de

Paulis

et

al.

Int

Arch

Allergy

Immunol

-LRB-

-RRB-

D.

D.

Taub

et

al.

J.

Clin

Invest

-LRB-

-RRB-

T.

W.

Klein

Nat

Rev.

Immunol

-LRB-

-RRB-

This

work

was

supported

by

grants

from

the

Deutsche

Forschungsgemeinschaft

-LSB-

and

-LRB-

to

M.K.

and

A.Z.

-RRB-

and

-LRB-

to

T.T.

-RRB-

-RSB-

by

a

Bonfor

stipend

to

E.G.

and

a

grant

from

Epitech

S.r.l.

to

V.D.M.

We

thank

M.

Krampert

for

her

help

in

wound

healing

experiments

L.

Cristino

for

her

help

with

immunohistochemistry

and

J.

Essig

A.

Zimmer

E.

Erxlebe

and

I.

Heim

for

technical

assistance

Supporting

Online

Material

Materials

and

Methods

Figs

to

Tables

and

References

March

accepted

May

science

Genome-Wide

Mapping

of

in

Vivo

Protein-DNA

Interactions

David

S.

Johnson

Ali

Mortazavi

Richard

M.

Myers

Barbara

In

vivo

protein-DNA

interactions

connect

each

transcription

factor

with

its

direct

targets

to

form

a

gene

network

scaffold

To

map

these

protein-DNA

interactions

comprehensively

across

entire

mammalian

genomes

we

developed

a

large-scale

chromatin

immunoprecipitation

assay

-LRB-

ChIPSeq

-RRB-

based

on

direct

ultrahigh-throughput

DNA

sequencing

This

sequence

census

method

was

then

used

to

map

in

vivo

binding

of

the

neuron-restrictive

silencer

factor

-LRB-

NRSF

also

known

as

REST

for

repressor

element

silencing

transcription

factor

-RRB-

to

locations

in

the

human

genome

The

data

display

sharp

resolution

of

binding

position

-LSB-

base

pairs

-LRB-

bp

-RRB-

-RSB-

which

facilitated

our

finding

motifs

and

allowed

us

to

identify

noncanonical

NRSF-binding

motifs

These

ChIPSeq

data

also

have

high

sensitivity

and

specificity

-LSB-

ROC

-LRB-

receiver

operator

characteristic

-RRB-

area

-RSB-

and

statistical

confidence

-LRB-

P

-RRB-

properties

that

were

important

for

inferring

new

candidate

interactions

These

include

key

transcription

factors

in

the

gene

network

that

regulates

pancreatic

islet

cell

development

A

lthough

much

is

known

about

transcription

factor

binding

and

action

at

specific

genes

far

less

is

known

about

the

composition

and

function

of

entire

factor-DNA

interactomes

especially

for

organisms

with

large

genomes

Now

that

human

mouse

and

other

large

genomes

have

been

sequenced

it

is

possible

in

principle

to

measure

how

any

transcription

factor

is

deployed

across

the

entire

genome

for

a

given

cell

type

and

physiological

condition

Such

measurements

are

important

for

systems-level

studies

because

they

provide

a

global

map

of

candidate

gene

network

input

connections

These

direct

physical

interactions

between

transcription

factors

or

cofactors

and

the

Department

of

Genetics

Stanford

University

School

of

Medicine

Stanford

CA

USA

Division

California

Institute

of

Technology

Pasadena

CA

USA

Institute

of

Technology

Beckman

Institute

Pasadena

CA

USA

These

authors

contributed

equally

to

this

work

To

whom

correspondence

should

be

addressed

E-mail

woldb@its.caltech.edu

-LRB-

B.W.

-RRB-

myers@shgc.stanford.edu

-LRB-

R.M.M.

-RRB-

chromosome

can

be

detected

by

chromatin

immunoprecipitation

-LRB-

ChIP

-RRB-

-LRB-

-RRB-

In

ChIP

experiments

an

immune

reagent

specific

for

a

DNA

binding

factor

is

used

to

enrich

target

DNA

sites

to

which

the

factor

was

bound

in

the

living

cell

The

enriched

DNA

sites

are

then

identified

and

quantified

For

the

gigabase-size

genomes

of

vertebrates

it

has

been

difficult

to

make

ChIP

measurements

that

combine

high

accuracy

whole-genome

completeness

and

high

binding-site

resolution

These

data-quality

and

depth

issues

dictate

whether

primary

gene

network

structure

can

be

inferred

with

reasonable

certainty

and

comprehensiveness

and

how

effectively

the

data

can

be

used

to

discover

binding-site

motifs

by

computational

methods

For

these

purposes

statistical

robustness

sampling

depth

across

the

genome

absolute

signal

and

signal-to-noise

ratio

must

be

good

enough

to

detect

nearly

all

in

vivo

binding

locations

for

a

regulator

with

minimal

inclusion

of

falsepositives

A

further

challenge

in

genomes

large

or

small

is

to

map

factor-binding

sites

with

high

positional

resolution

In

addition

to

making

comSCIENCE

VOL

putational

discovery

of

binding

motifs

feasible

this

dictates

the

quality

of

regulatory

site

annotation

relative

to

other

gene

anatomy

landmarks

such

as

transcription

start

sites

enhancers

introns

and

exons

and

conserved

noncoding

features

-LRB-

-RRB-

Finally

if

high-quality

protein-DNA

interactome

measurements

can

be

performed

routinely

and

at

reasonable

cost

it

will

open

the

way

to

detailed

studies

of

interactome

dynamics

in

response

to

specific

signaling

stimuli

or

genetic

mutations

To

address

these

issues

we

turned

to

ultrahigh-throughput

DNA

sequencing

to

gain

sampling

power

and

applied

size

selection

on

immuno-enriched

DNA

to

enhance

positional

resolution

The

ChIPSeq

assay

shown

here

differs

from

other

large-scale

ChIP

methods

such

as

ChIPArray

also

called

ChIPchip

-LRB-

-RRB-

ChIPSAGE

-LRB-

SACO

-RRB-

-LRB-

-RRB-

or

ChIPPet

-LRB-

-RRB-

in

design

data

produced

and

cost

The

design

is

simple

-LRB-

Fig.

-RRB-

and

unlike

SACO

or

ChIPPet

it

involves

no

plasmid

library

construction

Unlike

microarray

assays

the

vast

majority

of

single-copy

sites

in

the

genome

is

accessible

for

ChIPSeq

assay

-LRB-

-RRB-

rather

than

a

subset

selected

to

be

array

features

For

example

to

sample

with

similar

completeness

by

an

Affymetrix-style

microarray

design

a

nucleotide-by-nucleotide

sliding

window

design

of

roughly

billion

features

per

array

would

be

needed

for

the

nonrepeat

portion

of

the

human

genome

In

addition

ChIPSeq

counts

sequences

and

so

avoids

constraints

imposed

by

array

hybridization

chemistry

such

as

base

composition

constraints

related

to

Tm

the

temperature

at

which

of

double-stranded

DNA

or

DNA-RNA

hybrids

is

denatured

cross-hybridization

and

secondary

structure

interference

Finally

ChIPSeq

is

feasible

for

any

sequenced

genome

rather

than

being

restricted

to

species

for

which

wholegenome

tiling

arrays

have

been

produced

ChIPSeq

illustrates

the

power

of

new

sequencing

platforms

such

as

those

from

Solexa

Illumina

and

to

perform

sequence

census

counting

assays

The

generic

task

in

these

applications

is

to

identify

and

quantify

the

molecular

JUNE

Downloaded

from

on

May

REPORTS

contents

of

a

nucleic

acid

sample

whose

genome

of

origin

has

been

sequenced

The

very

large

numbers

of

short

individual

sequence

reads

produced

by

these

instruments

-LRB-

currently

reads

of

nucleotides

-LRB-

nt

-RRB-

or

million

reads

of

nt

per

instrument

run

depending

on

the

platform

used

-RRB-

are

extremely

well

suited

to

making

direct

digital

measurements

of

the

sequence

content

of

a

nucleic

acid

sample

By

determining

a

short

sequence

read

from

each

of

many

-LRB-

to

-RRB-

randomly

selected

molecules

from

the

sample

and

then

informatically

mapping

each

sequence

read

onto

the

reference

genome

the

identity

of

each

starting

molecule

is

learned

and

its

frequency

in

the

sample

is

calculated

Desired

levels

of

sensitivity

and

statistical

certainty

needed

to

detect

rare

molecular

species

can

be

achieved

in

principle

by

adjusting

the

total

number

of

sequence

reads

Sequence

census

assays

do

not

require

knowing

in

advance

that

a

sequence

is

of

interest

as

a

promoter

enhancer

or

RNA-coding

domain

as

most

current

microarray

designs

do

Below

we

use

the

Solexa/Illumina

platform

because

high-read

numbers

contribute

to

high

sensitivity

and

comprehensiveness

in

large

genomes

We

used

ChIPSeq

to

build

a

high-resolution

interactome

map

for

human

neuron-restrictive

silencer

factor

-LRB-

NRSF

also

known

as

REST

for

repressor

element

silencing

transcription

factor

-RRB-

This

zinc

finger

repressor

negatively

regulates

many

neuronal

genes

in

stem

and

progenitor

cells

and

in

nonneuronal

cell

types

such

as

the

Jurkat

T

cell

line

studied

here

-LRB-

-RRB-

A

primary

reason

for

selecting

NRSF

as

a

test

case

is

that

prior

studies

provide

a

large

set

of

known

goldstandard

target

genes

including

more

than

in

vivo

binding

sites

defined

by

ChIP-QPCR

-LRB-

quantitative

real-time

fluorescence

polymerase

chain

reaction

-RRB-

-LRB-

-RRB-

A

subset

of

these

genes

has

also

been

tested

for

regulatory

function

by

transfection

assays

-LRB-

-RRB-

In

addition

the

DNA

motif

bound

by

NRSF

called

NRSE

-LRB-

also

called

-RRB-

is

long

-LRB-

bp

-RRB-

and

well-specified

-LRB-

-RRB-

This

has

led

to

a

rich

group

of

computational

models

for

the

site

and

for

all

its

occurrences

in

the

human

genome

-LRB-

-RRB-

These

sites

provide

a

framework

of

explicit

predictions

that

can

now

be

tested

by

measuring

repressor

binding

globally

Finally

there

is

a

high-quality

monoclonal

antibody

-LRB-

-RRB-

that

recognizes

NRSF

efficiently

in

ChIP

experiments

-LRB-

-RRB-

We

prepared

two

DNA

samples

for

each

ChIPSeq

experiment

an

NRSF/REST

enriched

ChIP

sample

and

a

companion

control

sample

of

the

same

fixed

chromatin

but

without

immunoenrichment

In

an

effort

to

increase

positional

precision

and

to

provide

optimal

substrate

for

the

Solexa/Illumina

sequencing

platform

we

introduced

a

size-selection

step

after

cross-link

reversal

-LRB-

Fig.

and

fig

-RRB-

-LRB-

-RRB-

DNA

sequencing

of

each

sample

was

performed

by

the

Solexa/Illumina

protocol

-LRB-

-RRB-

Two

to

million

sequence

reads

were

produced

per

sample

of

which

about

half

mapped

to

single

sites

in

the

human

genome

-LRB-

table

-RRB-

Sequence

reads

that

map

to

multiple

sites

in

the

genome

were

removed

from

subsequent

analysis

This

eliminates

sequences

in

simple

repeats

some

complex

repeats

and

also

segments

that

are

not

unique

by

chance

The

location

of

each

remaining

unique

sequence

read

in

the

genome

was

recorded

To

accommodate

polymorphisms

relative

to

the

reference

genome

up

to

two

mismatches

were

allowed

The

resulting

sequence

read

distribution

was

processed

with

a

ChIPSeq

peak

locator

algorithm

developed

for

this

purpose

-LRB-

-RRB-

The

algorithm

finds

a

local

concentration

of

sequence

hits

-LRB-

a

location

cluster

-RRB-

and

within

that

location

calls

a

peak

We

then

required

of

these

a

minimum

fivefold

enrichment

of

sequence

reads

in

the

ChIP

sample

relative

to

the

corresponding

location

in

the

control

Fivefold

enrichment

is

a

Fig.

ChIPSeq

discovers

NRSF/REST

protein-DNA

binding

events

with

high

resolution

on

a

genome-wide

scale

-LRB-

A

-RRB-

Generalized

scheme

of

ChIPSeq

begins

with

ChIP

followed

by

size

selection

for

recovered

material

-LRB-

-RRB-

followed

by

standard

preparation

for

Solexa/Illumina

sequencing

An

optional

preamplification

step

after

immuno-enrichment

and

before

size

selection

can

be

inserted

to

work

from

smaller

cell

number

input

-LRB-

used

in

experiment

below

-RRB-

-LRB-

-RRB-

-LRB-

B

-RRB-

Close-up

view

of

ChIPSeq

data

mapped

to

a

novel

NRSF-binding

site

-LRB-

NRSE

black

arrowhead

-RRB-

located

in

the

gene

Experiment

-LRB-

no

preamplification

-RRB-

and

experiment

-LRB-

with

preamplification

-RRB-

Output

from

the

peak-call

algorithm

is

shown

for

this

locus

-LRB-

red

arrowhead

-RRB-

and

corresponds

closely

with

the

sole

NRSE

in

the

NEUROD

locus

-LRB-

black

arrowhead

-RRB-

conservative

choice

among

enrichment

thresholds

commonly

used

in

contemporary

largescale

ChIP

studies

A

location

that

passed

these

criteria

and

also

had

or

more

independent

sequence

reads

-LRB-

a

threshold

value

selected

based

on

the

sensitivity

and

specificity

analysis

described

below

-RRB-

was

called

an

NRSF-positive

binding

event

An

example

of

primary

ChIPSeq

data

from

two

independent

experiments

is

shown

in

Fig.

for

the

locus

This

positive

signal

which

has

intermediate

signal

intensity

and

statistical

certainty

-LRB-

P

-RRB-

identifies

a

novel

NRSF-binding

target

The

NRSF/REST

sequence-tag

distribution

centers

directly

over

the

only

canonical

NRSE

motif

in

a

region

which

is

located

in

the

open

reading

frame

of

the

gene

This

site

was

called

a

JUNE

VOL

SCIENCE

Downloaded

from

on

May

REPORTS

ChIPSeq

peak

by

the

locator

algorithm

-LSB-

open

source

available

at

-LRB-

-RRB-

-RSB-

A

previous

study

had

implicated

NRSF

in

repression

of

but

had

failed

to

find

a

local

site

computationally

The

authors

theorized

long-distance

repression

to

explain

the

effect

-LRB-

-RRB-

but

our

results

suggest

a

simpler

explanation

of

a

degenerate

site

within

the

gene

itself

Over

the

entire

primary

data

set

-LRB-

tables

and

-RRB-

the

distribution

of

sequence-tag

number

per

location

ranged

from

the

threshold

value

of

sequence

tags

to

a

maximum

of

tags

at

the

highest

signal

-LRB-

Fig.

and

fig

-RRB-

The

two

ChIPSeq

experiments

produced

similar

results

-LRB-

fig

-RRB-

mapping

shared

enriched

locations

most

of

which

occur

in

or

near

genes

NRSF-binding

sites

previously

identified

by

QPCR

or

transfection

assays

-LRB-

-RRB-

plus

a

set

of

known

negatives

-LRB-

-RRB-

were

used

to

measure

sensitivity

-LRB-

successful

detection

of

true

positives

-RRB-

and

specificity

-LRB-

successful

rejection

of

true

negatives

-RRB-

of

the

ChIPSeq

assay

A

ROC

-LRB-

receiver

operator

characteristic

-RRB-

analysis

provides

a

way

of

measuring

and

graphically

portraying

sensitivity

-LRB-

fraction

true

positive

on

the

y

axis

-RRB-

versus

specificity

-LRB-

the

fraction

of

false-positives

displayed

on

the

x

axis

-RRB-

-LRB-

Fig.

-RRB-

Perfect

sensitivity

and

specificity

would

produce

an

ROC

curve

that

traces

the

y

axis

to

a

value

of

which

would

extend

across

all

possible

x

values

to

produce

an

area

under

the

curve

of

In

contrast

entirely

random

classification

by

chance

would

produce

an

ROC

area

of

The

observed

ROC

areas

are

high

at

-LRB-

shown

-RRB-

and

for

the

two

independent

experiments

The

selected

threshold

of

sequence

reads

per

region

required

for

inclusion

in

the

ChIPSeq

interactome

corresponds

to

a

sensitivity

of

and

a

specificity

of

We

conclude

that

the

ChIPSeq

NRSF

interactome

measurements

are

accurate

and

as

suggested

by

P

values

-LRB-

table

-RRB-

statistically

robust

For

rough

comparison

a

recent

ChIPPet

study

of

the

interactome

did

not

measure

these

parameters

but

investigators

estimated

that

less

than

of

the

largest

group

of

positive

signals

-LRB-

sites

which

were

defined

by

two

paired

end

tags

-RRB-

are

true

sites

whereas

the

much

more

certain

and

smaller

class

-LRB-

sites

-RRB-

likely

misses

more

than

half

of

true

positives

-LRB-

-RRB-

In

general

we

expect

that

differences

in

immune

reagent

quality

epitope

availability

and

other

aspects

of

design

that

affect

all

ChIP

experiments

as

well

as

interactome

structure

itself

will

contribute

to

differences

between

studies

We

next

assessed

the

precision

of

ChIPSeq

site

location

relative

to

computationally

high-scoring

NRSE

motifs

in

the

genome

that

also

have

positive

ChIPSeq

signals

by

measuring

the

distance

from

the

experimental

ChIPSeq

peak

to

the

center

of

the

computational

NRSE

sequence

motif

In

this

group

sites

were

ChIPSeq-positive

in

both

experiments

and

the

center

of

a

NRSE

motif

was

within

bp

of

the

called

ChIPSeq

peak

-LRB-

-RRB-

for

of

these

-LRB-

Fig.

-RRB-

The

resolution

which

depends

in

part

on

size

selection

of

sheared

chromatin

after

immuno-enrichment

-LRB-

-RRB-

is

much

higher

than

is

typical

for

ChIPchip

or

ChIPSAGE

-LRB-

to

bp

-RRB-

-LRB-

-RRB-

How

comprehensive

are

the

NRSF

ChIPSeq

measurements

Several

lines

of

evidence

address

this

question

First

as

shown

in

Fig.

virtually

all

strong

canonical

NRSF

motif

instances

across

the

human

genome

were

detectably

occupied

We

defined

strong

sites

as

those

having

match

to

a

previously

developed

motif

model

-LRB-

a

position-specific

frequency

matrix

-RRB-

which

is

based

on

evolutionarily

conserved

site

instances

across

multiple

mammalian

genomes

-LRB-

-RRB-

This

high

representation

of

detectable

binding

suggests

that

no

strong

sites

were

missed

by

undersampling

It

also

implies

that

all

sites

are

accessible

for

NRSF/RESTbinding

in

Jurkat

cells

at

least

part-time

in

some

individual

cells

although

the

degree

of

accessibility

might

vary

and

may

account

for

wide

differences

in

the

number

of

tags

per

site

-LRB-

Fig.

-RRB-

Second

we

observed

ChIPSeq-positive

signals

for

sites

previously

studied

in

detail

by

transfection

analysis

-LRB-

-RRB-

and

they

correspond

to

a

wide

range

of

ChIPSeq

signals

with

all

but

one

scoring

positive

in

both

ChIPSeq

experiments

Taken

together

with

the

sensitivity

results

-LRB-

Fig.

-RRB-

these

observations

suggest

that

the

NRSF/REST

interactome

measurements

are

genome-comprehensive

and

have

been

sampled

deeply

enough

to

include

most

sites

known

by

any

other

criteria

to

be

biologically

active

even

if

relatively

weakly

This

level

of

genome

completeness

is

attributable

to

the

depth

of

Solexa

Illumina

sequence

sampling

and

is

substantially

greater

than

in

prior

studies

of

the

adenosine

monophosphate

-LRB-

cAMP

-RRB-

response

element

binding

protein

-LRB-

CREB

-RRB-

interactome

measured

by

SACO

-LRB-

-RRB-

and

the

interactome

measured

by

ChIPPet

-LRB-

-RRB-

Fig.

-LRB-

A

-RRB-

Histogram

of

all

ChIPSeq-positive

regions

as

a

function

of

sequence

read

number

that

map

within

that

region

zero

-LRB-

white

-RRB-

one

-LRB-

blue

-RRB-

or

two

or

more

-LRB-

red

-RRB-

canonical

motif

instances

defined

as

those

scoring

match

to

the

position-specific

frequency

matrix

-LRB-

PSFM

-RRB-

model

of

the

NRSF-binding

site

-LRB-

-RRB-

-LRB-

B

-RRB-

ROC

analysis

with

area

under

the

curve

for

experiment

-LRB-

shown

-RRB-

and

for

experiment

This

measures

the

performance

of

ChIPSeq

in

detecting

previously

validated

true

positives

-LRB-

-RRB-

and

true

negatives

-LRB-

-RRB-

as

described

in

the

text

and

-LRB-

-RRB-

The

threshold

used

for

subsequent

analysis

corresponds

to

and

specificity

as

indicated

by

the

arrowhead

ChIPSeq

false-negatives

corresponded

to

the

lowest

QPCR-positive

validation

values

-LRB-

C

-RRB-

Distribution

of

the

distance

from

the

center

of

high-scoring

canonical

NRSE

motifs

-LSB-

or

higher

match

to

the

PSFM

motif

model

-LRB-

-RRB-

-RSB-

in

ChIPSeq-enriched

regions

-LRB-

experiment

-RRB-

In

this

example

of

peaks

fall

within

the

boundaries

of

NRSE

-LRB-

here

to

bp

-RRB-

and

of

the

canonical

NRSEs

fall

within

bp

of

the

called

experimental

peak

-LRB-

D

-RRB-

Site

occupancy

detected

by

ChIPSeq

for

NRSE

motifs

in

Jurkat

cells

as

a

function

of

PSFM

score

-LRB-

-RRB-

SCIENCE

VOL

JUNE

Downloaded

from

on

May

REPORTS

We

next

asked

whether

NRSF

binding

in

or

near

promoters

is

correlated

with

low

levels

of

transcription

as

expected

for

a

transcriptional

repressor

To

answer

this

question

we

looked

for

high-confidence

promoter

predictions

-LRB-

-RRB-

that

occur

within

kb

of

a

ChIPSeq

peak

We

then

assessed

genome-wide

transcript

levels

in

Jurkat

cells

by

hybridizing

labeled

RNA

to

Illumina

Sentrix

arrays

The

subset

of

transcripts

corresponding

to

promoters

near

NRSF-binding

sites

had

significantly

lower

-LRB-

P

-RRB-

transcript

signals

than

the

full

set

of

transcripts

-LRB-

fig

-RRB-

This

argues

that

NRSF

binding

near

promoters

is

significantly

associated

with

transcriptional

repression

in

these

cells

The

positional

resolution

and

low

number

of

false-positives

in

these

experiments

can

greatly

facilitate

motif-finding

algorithms

The

effect

is

to

elevate

the

frequency

of

occurrence

of

true

motifs

within

the

input

DNA

relative

to

background

sequences

This

can

improve

signal-tonoise

and

also

greatly

reduce

the

run

times

for

many

algorithms

Much

is

known

about

the

canonical

NRSF-binding

site

-LRB-

NRSE

motif

-RRB-

-LRB-

-RRB-

and

this

allowed

us

to

ask

if

that

site

emerges

when

a

sample

of

the

experimental

interactome

peak

domains

are

used

We

first

applied

the

motif-finding

algorithm

MEME

-LRB-

-RRB-

to

all

sites

in

the

top

of

signal

intensity

-LRB-

segments

from

regions

having

reads

or

more

-RRB-

MEME

returned

the

full

previously

known

motif

-LRB-

table

-RRB-

Single

or

multiple

matches

to

this

canonical

motif

using

a

match

threshold

account

for

of

all

ChIPSeq

regions

mapped

in

this

study

We

next

focused

attention

on

those

remaining

ChIPSeq-positive

regions

that

have

or

more

ChIPSeq

reads

yet

have

no

canonical

motif

match

There

are

such

locations

and

when

they

were

run

in

MEME

only

two

candidate

motifs

stood

out

-LRB-

table

-RRB-

By

inspection

the

large

canonical

NRSF-binding

motif

of

bp

is

naturally

subdivided

into

two

prominent

nonidentical

nonpalindromic

half

sites

-LRB-

Fig.

-RRB-

The

two

motifs

from

the

MEME

search

correspond

directly

to

the

separate

left

and

right

sides

of

the

canonical

motif

We

next

asked

if

these

motifs

occur

at

other

ChIPSeq-binding

locations

and

if

they

are

organized

in

any

discernible

pattern

A

distinctive

pattern

was

discovered

within

bp

of

many

ChIPSeq

peaks

in

which

left

and

right

half-site

motifs

are

separated

by

additional

spacer

sequence

that

increases

the

center-to-center

distance

from

the

canonical

bp

to

to

bp

or

decreases

it

by

bp

to

bp

-LRB-

Fig.

and

fig

-RRB-

Thus

the

canonical

site

has

two

central

positions

that

have

no

sequence

specificity

and

the

noncanonical

group

is

similarly

oriented

but

has

increased

the

separation

distance

by

an

additional

to

bp

-LRB-

Fig.

-RRB-

These

linked

half

sites

oriented

with

respect

to

each

other

in

the

same

way

as

in

the

canonical

site

occur

in

NRSF

ChIPSeq

binding

domains

in

a

statistically

significant

manner

relative

to

random

sequence

windows

in

the

genome

-LRB-

for

the

half-site

distance

of

P

value

of

-RRB-

and

account

for

regions

lacking

a

canonical

motif

-LRB-

Fig.

and

fig

-RRB-

We

also

found

that

some

binding

locations

have

multiple

clustered

occurrences

of

noncanonical

motif

-LRB-

s

-RRB-

along

with

a

canonical

one

There

are

no

structural

data

available

for

NRSF

so

we

can

not

relate

this

new

family

of

binding-site

motifs

to

a

known

DNA

binding

structure

However

the

protein

has

eight

zinc

fingers

in

its

DNA

binding

domain

and

other

zinc

finger

proteins

such

as

bind

DNA

with

three

fingers

per

turn

but

they

show

considerable

strain

when

binding

with

six

fingers

-LRB-

-RRB-

This

makes

simultaneous

binding

of

one

molecule

of

NRSF

to

these

noncanonical

half-site

configurations

plausible

but

it

is

also

possible

that

the

protein

is

bound

to

only

one

half

site

at

a

time

by

using

a

subset

of

its

fingers

in

these

cases

It

will

be

interesting

to

learn

if

there

are

other

functional

and

molecular

characteristics

that

set

these

sites

apart

For

example

do

the

different

NRSF

co-repressors

differ

in

their

interactions

at

noncanonical

sites

compared

with

canonical

ones

-LRB-

-RRB-

We

also

asked

whether

half

sites

are

significantly

enriched

in

our

ChIPSeq

neighborhoods

without

regard

to

orientation

or

spacing

relative

to

expectations

based

on

their

occurrence

in

the

genomes

and

found

that

these

regions

are

greatly

enriched

for

left-side

half

sites

-LRB-

-RRB-

and

right-side

half

sites

-LRB-

-RRB-

This

range

of

configurations

from

concentrated

half

sites

to

the

noncanonical

spaced

left

and

right

sites

to

the

canonical

spaced

full

site

is

quite

striking

Significant

NRSF

binding

occurs

in

vivo

according

to

our

data

at

all

three

kinds

of

loci

Because

the

half

sites

are

much

shorter

than

the

full

NRSE

motif

they

also

occur

widely

over

the

genome

presumably

mainly

by

chance

This

would

mean

that

there

is

a

rich

pool

of

Fig.

-LRB-

A

-RRB-

Canonical

NRSF-binding

motif

WebLogo

-LRB-

-RRB-

Its

left

and

right

half

sites

have

a

center-to-center

distance

of

bp

-LRB-

B

-RRB-

Histogram

of

half-site

distances

in

ChIPSeq-enriched

regions

showing

the

observed

-LRB-

blue

-RRB-

and

expected

-LRB-

white

-RRB-

counts

-LRB-

based

on

frequency

in

the

genome

-RRB-

In

addition

to

the

expected

canonical

peak

at

distance

bp

there

is

also

significant

enrichment

of

half

sites

with

noncanonical

distances

of

to

bp

-LRB-

C

-RRB-

WebLogo

of

noncanonical

NRSE

with

half-site

distance

of

showing

the

lack

of

conservation

in

the

spacer

nucleotides

-LRB-

D

-RRB-

The

NRSFbinding

motifs

predicted

in

the

ChIPSeq-positive

regions

that

contain

canonical

noncanonical

or

only

half-site

motifs

JUNE

VOL

SCIENCE

Downloaded

from

on

May

REPORTS

possible

binding

sites

from

which

higher

affinity

canonical

sites

could

be

gradually

made

and

tested

in

evolution

as

suggested

previously

-LRB-

-RRB-

However

these

sites

were

considered

unlikely

to

interact

with

NRSF

specifically

-LRB-

-RRB-

whereas

within

the

noncanonical

motif

family

we

define

here

sites

bind

on

their

own

especially

when

clustered

-LRB-

fig

-RRB-

This

suggests

a

plausible

multistep

path

by

which

the

target-site

repertoire

could

evolve

beginning

with

clustered

partial

sites

passing

through

an

intermediate

and

more

specific

orientation

and

spacing

-LRB-

the

or

to

spaced

family

here

-RRB-

and

eventually

becoming

refined

into

the

canonical

site

-LRB-

the

spaced

classic

binding

motif

-RRB-

Because

there

are

more

than

multi

zinc

finger

transcription

factors

encoded

in

the

genome

-LRB-

-RRB-

many

of

which

are

evolving

rapidly

in

humans

and

mice

this

strategy

might

be

used

by

other

members

of

the

zinc

finger

family

We

found

that

genes

encoding

transcription

factors

microRNAs

and

five

splicing

regulators

were

occupied

by

NRSF

NRSEs

occur

prominently

in

introns

-LRB-

table

-RRB-

including

a

noncanonical

site

-LRB-

P

-RRB-

located

about

bp

downstream

of

the

transcription

start

site

of

the

NRSF

gene

itself

which

suggests

the

possibility

of

negative

autoregulatory

feedback

We

also

found

as

expected

that

NRSFbound

loci

are

highly

enriched

in

gene

ontology

-LRB-

GO

-RRB-

terms

related

to

neurons

and

their

development

-LRB-

Fig.

-RRB-

The

enrichment

for

the

experimentally

determined

sites

exceeded

that

achieved

for

any

computationally

predicted

target

gene

cohort

Synaptic

transmission

and

nervous

system

development

rank

in

the

top

three

GO

terms

among

with

P

values

for

overrepresentation

of

the

NRSF

target

genes

of

and

-LRB-

-RRB-

-LRB-

Fig.

-RRB-

This

group

includes

a

set

of

transcription

factors

that

have

not

previously

been

suggested

as

NRSF

targets

but

are

known

to

be

critical

in

the

gene

network

that

drives

islet

cell

development

in

the

pancreas

The

transcription

factors

hepatocyte

nuclear

factors

and

and

were

all

detected

here

for

the

first

time

as

in

vivo

binding

targets

of

NRSF

and

together

with

which

is

a

previously

identified

target

-LRB-

-RRB-

they

are

positioned

critically

in

the

regulatory

network

that

controls

pancreatic

b

cell

development

-LRB-

Fig.

-RRB-

-LRB-

-RRB-

Although

in

vivo

binding

does

not

ensure

NRSF

repression

activity

these

regulators

are

known

to

function

as

positive

drivers

of

pancreatic

neuroendocrine

development

If

NRSF

repression

is

active

at

all

these

sites

as

might

be

the

case

in

progenitor

cells

the

circuit

would

be

effectively

blocked

In

this

hypothesis

NRSF

acts

as

a

permissivity

factor

gating

entry

into

and

progress

through

the

developmental

pathway

These

pancreatic

network

sites

are

among

the

more

modest

ChIPSeq

signals

ranging

from

sequence

reads

for

to

sequence

reads

for

values

that

are

comfortably

above

the

significance

threshold

of

-LRB-

set

on

the

basis

of

sensitivity/specificity

considerations

and

known

regulatory

targets

of

Fig.

A

and

B

-RRB-

yet

they

fall

in

the

bottom

quartile

Thus

these

ChIPSeq

data

were

statistically

robust

enough

to

map

parts

of

this

gene

network

that

might

otherwise

have

gone

undetected

or

been

highly

uncertain

-LRB-

Fig.

and

fig

-RRB-

There

are

precedents

in

other

systems

that

show

that

relatively

weak

sites

are

biologically

important

specifically

because

they

are

in

the

biochemical

binding

sense

suboptimal

For

example

in

Caenorhabditis

elegans

the

factor

is

the

key

activator

of

a

large

interactome

and

a

subset

of

target

genes

has

suboptimal

sequences

and

numbers

of

sites

-LRB-

-RRB-

In

that

system

when

binding

is

suboptimal

it

is

believed

Fig.

-LRB-

A

-RRB-

Gene

ontology

-LRB-

GO

-RRB-

analysis

of

the

computationally

predicted

cohort

of

NRSF

target

genes

-LSB-

genes

scoring

as

match

to

the

previously

developed

computational

model

for

the

NRSE

motif

-LRB-

-RRB-

-RSB-

compared

with

ChIPSeq-positive

genes

-LRB-

right

-RRB-

P

values

for

enrichment

of

GO

terms

above

the

significance

threshold

of

which

accounts

for

multiple

hypothesis

testing

-LRB-

-RRB-

are

indicated

by

the

color

scale

GO

terms

below

the

significance

threshold

are

in

white

boxes

ChIPSeq

NRSE

target

genes

are

most

enriched

in

synaptic

transmission

nervous

system

development

and

ion

channel-activity

functions

-LRB-

tables

and

-RRB-

-LRB-

B

-RRB-

ChIPSeq

data

identified

new

candidate

connections

-LRB-

blue

-RRB-

between

NRSF

and

members

of

the

pancreatic

islet

b

cell

specification

gene

regulatory

network

-LSB-

adapted

from

-LRB-

-RRB-

-RSB-

Key

transcription

factors

bound

by

NRSF

including

and

occupy

positions

high

in

the

network

that

govern

network

activation

and

progression

ChIPSeq

data

also

confirmed

previously

known

NRSF

targets

-LRB-

cyan

-RRB-

that

include

terminal

differentiation

genes

such

as

-LRB-

-RRB-

SCIENCE

VOL

JUNE

Downloaded

from

on

May

REPORTS

to

help

program

the

temporal

order

of

action

during

development

with

poor

binders

turning

on

at

later

times

in

the

developmental

progression

when

levels

are

highest

By

analogy

the

regulators

that

govern

the

pancreatic

network

may

be

released

from

NRSF

repression

relatively

early

in

down-regulation

of

the

repressor

to

create

a

permissive

state

that

must

be

established

before

the

neuroendocrine

development

program

is

launched

Also

following

this

logic

is

a

classic

NRSF

target

gene

that

is

expressed

later

in

development

in

differentiated

islet

cells

and

it

displayed

relatively

higher

ChIPSeq

tag

scores

than

most

of

the

transcription

factors

that

are

positioned

higher

and

earlier

in

the

network

Independent

evidence

suggests

that

expression

depends

on

relief

from

NRSF-mediated

repression

in

islet

cells

-LRB-

-RRB-

Targets

of

the

regulatory

class

highlighted

here

-LRB-

Fig.

-RRB-

can

also

participate

in

positive

autoregulatory

and

cross-regulatory

interactions

that

we

expect

would

stabilize

and

push

forward

the

circuit

once

it

begins

-LRB-

-RRB-

This

makes

a

protective

repressor

active

in

nonpancreatic

cell

types

or

progenitor

cells

an

attractive

piece

of

regulatory

logic

The

initial

picture

we

have

of

the

experimentally

determined

NRSF/REST

interactome

has

been

drawn

for

one

cell

type

-LRB-

Jurkat

T

cell

line

-RRB-

and

T

cells

express

this

factor

at

relatively

high

levels

In

this

cell

environment

the

interactome

is

composed

of

three

broad

classes

of

target

loci

with

respect

to

binding

motifs

First

are

loci

defined

by

near-optimal

canonical

motifs

and

virtually

all

of

these

bound

the

factor

detectably

in

vivo

This

suggests

that

for

biochemically

optimal

sites

there

is

sufficient

chromatin

access

and

high

enough

DNA

affinity

to

establish

measurable

occupancy

although

the

strength

of

the

ChIPSeq

signal

among

these

sites

varied

over

wide

range

Second

are

loci

containing

instances

of

the

noncanonical

motif

family

-LRB-

Fig.

-RRB-

or

sites

that

are

weaker

matches

to

the

canonical

site

Binding

of

sites

in

this

class

could

not

be

predicted

solely

on

the

basis

of

their

motif

sequence

Several

hundred

instances

showed

significant

binding

whereas

thousands

of

others

in

this

group

had

no

detectable

ChIPSeq

signal

Binding-motif

properties

we

do

not

yet

appreciate

differential

chromatin

access

or

epitope

exposure

for

subset

of

NRSF-containing

complexes

might

discriminate

the

minority

ChIP-positives

in

this

group

from

the

majority

that

are

nonbinding

These

observations

argue

that

global

experimental

data

are

needed

to

discriminate

motif

instances

occupied

in

vivo

from

many

others

that

appear

similar

in

motif

quality

but

are

not

similarly

occupied

even

for

a

factor

like

NRSF/REST

which

has

a

well-specified

binding-motif

family

Finally

there

is

a

third

small

but

interesting

class

that

binds

NRSF/REST

reproducibly

and

in

some

cases

quite

robustly

but

lacks

any

identifiable

NRSE

motif

except

for

the

presence

of

half

sites

-LRB-

tables

and

-RRB-

It

is

uncertain

if

these

are

explained

by

their

half-site

content

because

they

make

up

a

tiny

minority

of

loci

with

superficially

similar

half-site

content

This

raises

the

possibility

that

NRSF/REST

might

associate

indirectly

rather

than

directly

with

a

limited

number

of

specific

chromosomal

locations

ChIPSeq

as

performed

here

is

relatively

cost-effective

Solexa/Illumina

platform

sequencing

costs

per

experiment

are

currently

about

half

that

of

the

most

comprehensive

human

wholegenome

tiling

arrays

ChIPSeq

sampling

that

is

to

times

deeper

than

was

used

here

is

plausible

now

within

the

general

range

of

microarray

costs

and

this

capacity

may

be

needed

for

interactomes

having

many

more

sites

per

genome

than

NRSF

For

example

it

is

possible

that

some

widely

used

transcription

cofactors

or

chromatin

remodeling

complexes

might

have

on

the

order

of

to

true-positive

sites

distributed

over

a

wide

dynamic

range

of

occupancy

levels

and

such

interactome

structures

will

require

correspondingly

deeper

sequence

sampling

Other

ultrahigh-throughput

sequencing

platforms

such

as

the

one

from

Life

Sciences

could

also

be

used

to

assay

ChIP

products

but

whatever

sequencing

platform

is

used

our

results

indicate

that

read

number

capacity

and

input

ChIP

DNA

size

are

key

parameters

References

and

Notes

T.

H.

Kim

B.

Ren

Annu

Rev.

Genom

Hum

Genet

-LRB-

-RRB-

ENCODE

Project

Consortium

Science

-LRB-

-RRB-

S.

Impey

et

al.

Cell

-LRB-

-RRB-

C.

L.

Wei

et

al.

Cell

-LRB-

-RRB-

Materials

and

methods

are

available

as

supporting

material

on

Science

Online

N.

Ballas

G.

Mandel

Curr

Opin

Neurobiol

-LRB-

-RRB-

A.

Mortazavi

E.

C.

Thompson

S.

T.

Garcia

R.

M.

Myers

B.

Wold

Genome

Res

-LRB-

-RRB-

C.

J.

Schoenherr

A.

J.

Paquette

D.

J.

Anderson

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

A.

W.

Bruce

et

al.

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

C.

Zhang

et

al.

Nucleic

Acids

Res

-LRB-

-RRB-

J.

Wu

X.

Xie

Genome

Biol

-LRB-

-RRB-

Z.

F.

Chen

A.

J.

Paquette

D.

J.

Anderson

Nat

Genet

-LRB-

-RRB-

Solexa

sequencing

technology

ilmn?ID

ChIPSeq

peak

finder

available

at

V.

V.

Lunyak

et

al.

Science

-LRB-

-RRB-

S.

Cawley

et

al.

Cell

-LRB-

-RRB-

N.

C.

Jones

P.

A.

Pevzner

Bioinformatics

-LRB-

-RRB-

T.

L.

Bailey

C.

Elkan

Mach

Learn

-LRB-

-RRB-

E.

Peisach

C.

O.

Pabo

J.

Mol

Biol

-LRB-

-RRB-

N.

Ballas

C.

Grunseich

D.

D.

Lu

J.

C.

Speh

G.

Mandel

Cell

-LRB-

-RRB-

M.

Yeo

et

al.

Science

-LRB-

-RRB-

S.

Huntley

et

al.

Genome

Res

-LRB-

-RRB-

E.

H.

Davidson

The

Regulatory

Genome

Gene

Regulatory

Networks

in

Development

and

Evolution

-LRB-

Academic

Press

Elsevier

San

Diego

CA

-RRB-

J.

Gaudet

S.

E.

Mango

Science

-LRB-

-RRB-

F.

Atouf

P.

Czernichow

R.

Scharfmann

J.

Biol

Chem

-LRB-

-RRB-

Web-based

sequence

logo

generating

application

Weblogo.berkeley.edu

We

thank

G.

Schroth

and

his

group

at

Solexa/Illumina

Inc.

for

access

to

the

sequencing

platform

without

which

this

work

would

not

have

been

possible

We

thank

B.

Anton

L.

Nguyen

C.

Medina

and

L.

Tsavaler

for

outstanding

experimental

work

and

K.

F.

McCue

for

invaluable

counsel

on

statistical

analyses

We

are

grateful

to

D.

Anderson

of

Caltech

for

the

gift

of

monoclonal

antibody

against

NRSF

This

work

was

supported

by

NIH

grant

to

R.M.M.

with

a

supplement

to

B.W.

and

by

a

grant

from

the

Caltech

Beckman

Institute

A.M.

was

supported

by

NIH/National

Research

Service

Award

B.W.

by

a

Bren

Chair

endowment

at

Caltech

and

R.M.M.

by

the

Stanford

W.

Ascherman

Chair

endowment

at

Stanford

University

Supporting

Online

Material

Materials

and

Methods

Figs

to

Tables

to

References

February

accepted

April

Published

online

May

science

Include

this

information

when

citing

this

paper

JUNE

VOL

SCIENCE

Downloaded

from

on

May

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Vol

October

doi

LETTERS

Quantifying

the

evolutionary

dynamics

of

language

Erez

Jean-Baptiste

Joe

Tina

Martin

A.

Human

language

is

based

on

grammatical

Cultural

evolution

allows

these

rules

to

change

over

Rules

compete

with

each

other

as

new

rules

rise

to

prominence

old

ones

die

away

To

quantify

the

dynamics

of

language

evolution

we

studied

the

regularization

of

English

verbs

over

the

past

years

Although

an

elaborate

system

of

productive

conjugations

existed

in

English

's

proto-Germanic

ancestor

Modern

English

uses

the

dental

suffix

ed

to

signify

past

Here

we

describe

the

emergence

of

this

linguistic

rule

amidst

the

evolutionary

decay

of

its

exceptions

known

to

us

as

irregular

verbs

We

have

generated

a

data

set

of

verbs

whose

conjugations

have

been

evolving

for

more

than

a

millennium

tracking

inflectional

changes

to

Old-English

irregular

verbs

Of

these

irregular

verbs

remained

irregular

in

Middle

English

and

are

still

irregular

today

We

study

how

the

rate

of

regularization

depends

on

the

frequency

of

word

usage

The

half-life

of

an

irregular

verb

scales

as

the

square

root

of

its

usage

frequency

a

verb

that

is

times

less

frequent

regularizes

times

as

fast

Our

study

provides

a

quantitative

analysis

of

the

regularization

process

by

which

ancestral

forms

gradually

yield

to

an

emerging

linguistic

rule

Natural

languages

comprise

elaborate

systems

of

rules

that

enable

one

speaker

to

communicate

with

These

rules

serve

to

simplify

the

production

of

language

and

enable

an

infinite

array

of

comprehensible

However

each

rule

has

exceptions

and

even

the

rules

themselves

wax

and

wane

over

centuries

and

Verbs

that

obey

standard

rules

of

conjugation

in

their

native

language

are

called

regular

In

the

Modern

English

language

regular

verbs

are

conjugated

into

the

simple

past

and

past-participle

forms

by

appending

the

dental

suffix

ed

to

the

root

-LRB-

for

instance

infinitive/simple

past/past

participle

talk/talked

talked

-RRB-

Irregular

verbs

obey

antiquated

rules

-LRB-

sing/sang/sung

-RRB-

or

in

some

cases

no

rule

at

all

-LRB-

go/went

-RRB-

New

verbs

entering

English

universally

obey

the

regular

conjugation

-LRB-

google/googled/googled

-RRB-

and

many

irregular

verbs

eventually

regularize

It

is

much

rarer

for

regular

verbs

to

become

irregular

for

every

sneak

that

snuck

there

are

many

more

flews

that

flied

out

Although

less

than

of

modern

verbs

are

irregular

the

ten

most

common

verbs

are

all

irregular

-LRB-

be

have

do

go

say

can

will

see

take

get

-RRB-

The

irregular

verbs

are

heavily

biased

towards

high

frequencies

of

Linguists

have

suggested

an

evolutionary

hypothesis

underlying

the

frequency

distribution

of

irregular

verbs

uncommon

irregular

verbs

tend

to

disappear

more

rapidly

because

they

are

less

readily

learned

and

more

rapidly

To

study

this

phenomenon

quantitatively

we

studied

verb

inflection

beginning

with

Old

English

-LRB-

the

language

of

Beowulf

spoken

around

AD

-RRB-

continuing

through

Middle

English

-LRB-

the

language

of

Chaucer

's

Canterbury

Tales

spoken

around

AD

-RRB-

and

ending

with

Modern

English

the

language

as

it

is

spoken

today

The

modern

ed

rule

descends

from

Old

English

weak

conjugation

which

applied

to

three-quarters

of

all

Old

English

The

exceptions

ancestors

of

the

modern

irregular

verbs

were

mostly

members

of

the

so-called

strong

verbs

There

are

seven

different

classes

of

strong

verbs

with

exemplars

among

the

Modern

English

irregular

verbs

each

with

distinguishing

markers

that

often

include

characteristic

vowel

shifts

Although

stable

coexistence

of

multiple

rules

is

one

possible

outcome

of

rule

dynamics

this

is

not

what

occurred

in

English

verb

We

therefore

define

regularity

with

respect

to

the

modern

ed

rule

and

call

all

these

exceptional

forms

irregular

We

consulted

a

large

collection

of

grammar

textbooks

describing

verb

inflection

in

these

earlier

epochs

and

hand-annotated

every

irregular

verb

they

described

-LRB-

see

Supplementary

Information

-RRB-

This

provided

us

with

a

list

of

irregular

verbs

from

ancestral

forms

of

English

By

eliminating

verbs

that

were

no

longer

part

of

Modern

English

we

compiled

a

list

of

Old

English

irregular

verbs

that

remain

part

of

the

language

to

this

day

Of

these

Old

English

irregulars

remained

irregular

in

Middle

English

and

are

still

irregular

in

Modern

English

Verbs

such

as

help

grip

and

laugh

which

were

once

irregular

have

become

regular

with

the

passing

of

time

Next

we

obtained

frequency

data

for

all

verbs

by

using

the

CELEX

corpus

which

contains

million

words

from

a

wide

variety

of

textual

For

each

of

our

verbs

we

calculated

the

frequency

of

occurrence

among

all

verbs

We

subdivided

the

frequency

spectrum

into

six

logarithmically

spaced

bins

from

to

Figure

shows

the

number

of

irregular

verbs

in

each

frequency

bin

There

are

only

two

verbs

be

and

have

in

the

highest

frequency

bin

with

mean

frequency

Both

remain

irregular

to

the

present

day

There

are

irregular

verbs

in

the

second

bin

with

mean

frequency

between

and

These

verbs

have

all

remained

irregular

from

Old

English

to

Modern

English

In

the

third

bin

with

a

mean

frequency

between

and

we

find

that

irregular

verbs

of

Old

English

all

remained

irregular

in

Middle

English

but

only

of

them

are

irregular

in

Modern

English

Four

verbs

in

this

frequency

range

help

reach

walk

and

work

underwent

regularization

In

the

fourth

frequency

bin

to

irregular

verbs

of

Old

English

have

left

in

Middle

and

in

Modern

English

In

the

fifth

frequency

bin

to

irregulars

of

Old

English

have

left

in

Middle

and

in

Modern

English

In

the

sixth

frequency

bin

to

irregulars

of

Old

English

decline

to

in

Middle

and

only

in

Modern

English

slink

a

verb

that

aptly

describes

this

quiet

process

of

disappearance

-LRB-

Table

-RRB-

Plotting

the

number

of

irregular

verbs

against

their

frequency

generates

a

unimodal

distribution

with

a

peak

between

and

This

unimodal

distribution

again

demonstrates

that

irregular

verbs

are

not

an

arbitrary

subset

of

all

verbs

because

a

random

subset

of

verbs

-LRB-

such

as

all

verbs

that

contain

the

letter

m

-RRB-

would

follow

Zipf

's

law

a

power

law

with

a

slope

of

-LRB-

refs

-RRB-

Program

for

Evolutionary

Dynamics

Department

of

Organismic

and

Evolutionary

Biology

Department

of

Mathematics

of

Applied

Mathematics

Harvard

University

Cambridge

Massachusetts

USA

Division

of

Health

Sciences

and

Technology

Massachusetts

Institute

of

Technology

Cambridge

Massachusetts

USA

Department

of

Systems

Biology

Harvard

Medical

School

Boston

Massachusetts

USA

These

authors

contributed

equally

to

this

work

Nature

Publishing

Group

LETTERS

NATURE

Vol

October

a

Number

of

irregular

verbs

Old

English

Middle

English

Modern

English

Frequency

b

Regularization

rate

Old

to

Modern

English

Middle

to

Modern

English

Frequency

Figure

Irregular

verbs

regularize

at

a

rate

that

is

inversely

proportional

to

the

square

root

of

their

usage

frequency

a

The

evolution

of

verbs

from

Old

English

-LRB-

green

-RRB-

over

time

through

Middle

English

-LRB-

red

-RRB-

and

Modern

English

-LRB-

blue

-RRB-

The

fraction

remaining

irregular

in

each

bin

decreases

as

the

frequency

decreases

The

frequency

shown

is

that

of

the

modern

descendant

and

was

computed

using

the

CELEX

corpus

Error

bars

indicate

standard

deviation

and

were

calculated

using

the

bootstrap

method

b

The

regularization

rate

of

irregular

verbs

as

a

function

of

frequency

The

relative

regularization

rates

obtained

by

comparing

Old

versus

Modern

English

-LRB-

green

-RRB-

and

Middle

versus

Modern

English

-LRB-

red

-RRB-

scale

linearly

on

a

log

log

plot

with

a

downward

slope

of

nearly

one-half

The

regularization

rate

and

the

half-life

scale

with

the

square

root

of

the

frequency

Four

of

our

six

frequency

bins

those

between

and

allow

us

to

estimate

the

relative

regularization

rates

of

irregular

verbs

Calculating

the

relative

regularization

rates

of

verbs

of

different

Table

The

irregular

verbs

studied

Frequency

Verbs

frequencies

is

independent

of

time

which

makes

the

dating

of

Old

and

Middle

English

irrelevant

for

this

calculation

We

can

plot

regularization

rate

versus

frequency

and

fit

a

straight

line

in

a

log

log

plot

-LRB-

Fig.

-RRB-

By

comparing

Old

and

Modern

English

we

obtain

a

slope

of

about

Therefore

an

irregular

verb

that

is

times

less

frequent

is

regularized

times

as

fast

In

other

words

the

half-life

of

irregular

verbs

is

proportional

to

the

square

root

of

their

frequency

By

comparing

Middle

and

Modern

English

we

find

a

slope

of

about

consistent

with

the

previous

result

Both

comparisons

show

that

low-frequency

irregular

verbs

are

selectively

forgotten

Figure

shows

the

exponential

decay

of

the

irregular

verbs

in

the

four

frequency

bins

between

and

as

a

function

of

time

From

these

data

which

depend

on

the

dating

of

Old

and

Middle

English

we

can

estimate

actual

half-lives

of

the

irregular

verbs

in

different

frequency

bins

Irregular

verbs

that

occur

with

a

frequency

between

and

have

a

half-life

of

about

years

whereas

those

with

a

frequency

between

and

have

a

half-life

of

years

If

we

fit

half-life

versus

frequency

with

a

straight

line

in

a

log

log

plot

we

obtain

a

slope

of

which

again

suggests

that

the

half-life

of

irregular

verbs

is

proportional

to

approximately

the

square

root

of

their

frequency

-LRB-

Fig.

-RRB-

It

is

noteworthy

that

various

methods

of

fitting

the

data

give

the

same

results

We

can

not

directly

determine

the

regularization

rate

for

frequency

bins

above

because

regularization

is

so

slow

that

no

event

was

observed

in

the

time

span

of

our

data

however

we

can

extrapolate

For

instance

the

half-life

of

verbs

with

frequencies

between

and

should

be

years

For

these

bins

the

population

is

so

small

and

the

half-life

so

long

that

we

may

not

see

a

regularization

event

in

the

lifetime

of

the

English

language

To

test

whether

the

dynamics

within

individual

competing

rules

were

captured

by

our

global

analysis

we

studied

the

decay

of

individual

classes

of

strong

verbs

-LRB-

for

example

hit/hit/hit

hurt/hurt

hurt

draw/drew/drawn

grow/grew/grown

-RRB-

Although

our

resolution

is

limited

by

the

small

sample

size

exponential

decay

is

once

again

observed

with

similar

exponents

-LRB-

see

Supplementary

Fig.

-RRB-

Like

a

Cheshire

cat

dying

rules

vanish

one

instance

at

a

time

leaving

behind

a

unimodal

frown

Because

adequate

corpora

of

Old

and

Middle

English

do

not

exist

we

have

estimated

the

frequency

of

an

irregular

verb

of

Old

and

Middle

English

by

the

frequency

of

the

corresponding

-LRB-

regular

or

irregular

-RRB-

verb

of

Modern

A

large

fraction

of

verbs

would

have

had

to

change

frequency

by

several

orders

of

magnitude

to

Regularization

-LRB-

-RRB-

Half-life

-LRB-

yr

-RRB-

be

have

come

do

find

get

give

go

know

say

see

take

think

begin

break

bring

buy

choose

draw

drink

drive

eat

fall

fight

forget

grow

hang

help

hold

leave

let

lie

lose

reach

rise

run

seek

set

shake

sit

sleep

speak

stand

teach

throw

understand

walk

win

work

write

arise

bake

bear

beat

bind

bite

blow

bow

burn

burst

carve

chew

climb

cling

creep

dare

dig

drag

flee

float

flow

fly

fold

freeze

grind

leap

lend

lock

melt

reckon

ride

rush

shape

shine

shoot

shrink

sigh

sing

sink

slide

slip

smoke

spin

spring

starve

steal

step

stretch

strike

stroke

suck

swallow

swear

sweep

swim

swing

tear

wake

wash

weave

weep

weigh

wind

yell

yield

bark

bellow

bid

blend

braid

brew

cleave

cringe

crow

dive

drip

fare

fret

glide

gnaw

grip

heave

knead

low

milk

mourn

mow

prescribe

redden

reek

row

scrape

seethe

shear

shed

shove

slay

slit

smite

sow

span

spurn

sting

stink

strew

stride

swell

tread

uproot

wade

warp

wax

wield

wring

writhe

bide

chide

delve

flay

hew

rue

shrive

slink

snip

spew

sup

wreak

Old

English

irregular

verbs

were

compiled

for

this

study

These

are

arranged

according

to

frequency

bin

and

in

alphabetical

order

within

each

bin

Also

shown

is

the

percentage

of

verbs

in

each

bin

that

have

regularized

The

half-life

is

shown

in

years

Verbs

that

have

regularized

are

indicated

in

red

As

we

move

down

the

list

an

increasingly

large

fraction

of

the

verbs

are

red

the

frequencydependent

regularization

of

irregular

verbs

becomes

immediately

apparent

Nature

Publishing

Group

NATURE

Vol

October

LETTERS

a

Number

of

irregular

verbs

Tim

e

-LRB-

ye

ars

AD

-RRB-

Number

of

irregular

verbs

Regularization

rate

Slope

Frequency

Frequency

Figure

Extrapolating

forward

and

backward

in

time

using

the

observation

that

regularization

rate

scales

as

the

square

root

of

frequency

The

differential

system

is

exactly

solvable

and

the

solution

fits

all

three

observed

distributions

As

we

move

backward

in

time

the

distribution

of

irregular

verbs

approaches

the

zipfian

distribution

characteristic

of

random

sets

of

words

The

distribution

for

exceptions

to

the

ed

rule

became

nonrandom

because

of

frequency-dependent

regularization

due

to

selective

pressure

from

the

emerging

rule

Figure

Irregular

verbs

decay

exponentially

over

time

a

Specifying

approximate

dates

of

Old

and

Middle

English

allows

computation

of

absolute

regularization

rates

Regularization

rates

increase

as

frequencies

decrease

but

are

otherwise

constant

over

time

b

Absolute

rates

of

regularization

are

shown

as

a

function

of

frequency

Error

bars

indicate

standard

deviation

and

were

calculated

using

the

bootstrap

method

The

square-root

scaling

is

obtained

again

interfere

with

the

effects

observed

To

verify

that

large

changes

in

frequency

are

rare

we

compared

frequency

data

from

CELEX

with

frequencies

drawn

from

the

largest

available

corpus

of

Middle

English

Of

verbs

only

had

frequency

changes

greater

than

a

factor

of

-LRB-

Supplementary

Fig.

-RRB-

Our

analysis

covers

a

vast

period

spanning

the

Norman

invasion

and

the

invention

of

the

printing

press

but

these

events

did

not

upset

the

dynamics

of

English

regularization

Therefore

it

is

possible

to

retrospectively

trace

the

evolution

of

the

irregular

verbs

moving

backwards

in

time

from

the

observed

Modern

English

distribution

and

up

through

Middle

and

Old

English

Going

still

further

back

in

time

allows

us

to

explore

the

effects

of

completely

undoing

the

frequency-dependent

selective

process

that

the

irregular

verbs

have

undergone

Eventually

the

shape

of

the

curve

changes

from

unimodal

to

a

power

law

decline

with

slope

of

nearly

-LRB-

Fig.

-RRB-

This

finding

is

consistent

with

the

fact

that

random

subsets

of

verbs

-LRB-

and

of

all

types

of

words

-RRB-

exhibit

such

a

zipfian

distribution

The

observed

irregular

verb

distribution

is

the

result

of

selective

pressure

on

a

random

collection

of

ancestral

verbs

We

can

also

make

predictions

about

the

future

of

the

past

tense

By

the

time

one

verb

from

the

set

begin

break

bring

buy

choose

draw

drink

drive

eat

fall

will

regularize

five

verbs

from

the

set

bid

dive

heave

shear

shed

slay

slit

sow

sting

stink

will

be

regularized

If

the

current

trends

continue

only

of

the

verbs

studied

will

be

irregular

in

What

will

be

the

next

irregular

verb

to

regularize

It

is

likely

to

be

wed/wed/wed

The

frequency

of

wed

is

only

uses

per

million

verbs

ranking

at

the

very

bottom

of

the

modern

irregular

verbs

Indeed

it

is

already

being

replaced

in

many

contexts

by

wed

wedded/wedded

Now

is

your

last

chance

to

be

a

newly

wed

The

married

couples

of

the

future

can

only

hope

for

wedded

bliss

In

previous

millennia

many

rules

vied

for

control

of

English

language

conjugation

and

fossils

of

those

rules

remain

to

this

day

Yet

from

this

primordial

soup

of

conjugations

the

dental

suffix

ed

emerged

triumphant

The

competing

rules

are

long

dead

and

unfamiliar

even

to

well-educated

native

speakers

These

rules

disappeared

because

of

the

gradual

erosion

of

their

instances

by

a

process

that

we

call

regularization

But

regularity

is

not

the

default

state

of

a

language

a

rule

is

the

tombstone

of

a

thousand

exceptions

METHODS

SUMMARY

We

searched

reference

works

on

Old

and

Middle

English

compiling

a

list

of

every

irregular

verb

that

we

found

We

determined

whether

each

verb

is

still

present

in

Modern

English

For

all

Old

English

verbs

whose

descendants

remained

in

the

English

language

we

checked

whether

they

were

still

irregular

using

a

complete

listing

of

the

modern

irregular

verbs

If

they

had

regularized

we

determined

when

regularization

had

occurred

on

the

basis

of

the

last

time

period

in

which

we

found

a

positive

annotation

listing

the

verb

as

irregular

A

list

of

sources

used

and

the

entire

resulting

annotation

are

provided

in

the

Supplementary

Information

We

determined

usage

frequencies

for

all

the

verbs

using

the

CELEX

database

We

then

binned

the

Old

English

irregular

verbs

using

a

standard

logarithmic

binning

algorithm

in

Python

We

used

the

resulting

binning

to

determine

regularization

rates

for

verbs

of

differing

frequencies

Regularization

rates

-LRB-

Fig.

-RRB-

for

each

bin

were

computed

directly

The

fits

to

exponential

decay

-LRB-

Fig.

-RRB-

and

to

the

solution

of

the

irregular

equation

-LRB-

Fig.

and

Supplementary

Information

-RRB-

were

produced

using

the

method

of

least

squares

The

Python

source

code

for

producing

the

figures

and

the

table

is

available

at

Received

March

accepted

July

Chomsky

N.

Aspects

of

the

Theory

of

Syntax

-LRB-

MIT

Press

Cambridge

-RRB-

Lightfoot

D

The

Development

of

Language

Acquisition

Change

and

Evolution

-LRB-

Blackwell

Oxford

-RRB-

Clark

R.

Roberts

I

A

computational

model

of

language

learnability

and

language

change

Linguist

Inq

-LRB-

-RRB-

Abrams

D.

Strogatz

S.

Modelling

the

dynamics

of

language

death

Nature

-LRB-

-RRB-

Nowak

M.

A.

Komarova

N.

L.

Niyogi

P.

Computational

and

evolutionary

aspects

of

language

Nature

-LRB-

-RRB-

Tim

e

-LRB-

y

b

800 1200 2000

ear

sA

D

-RRB-

uency

Freq

Nature

Publishing

Group

LETTERS

NATURE

Vol

October

Hooper

J.

in

Current

Progress

in

Historical

Linguistics

-LRB-

ed

Christie

W.

-RRB-

-LRB-

North-Holland

Amsterdam

-RRB-

Hauser

M.

D.

Chomsky

N.

Fitch

W.

T

The

faculty

of

language

what

is

it

who

has

it

and

how

did

it

evolve

Science

-LRB-

-RRB-

Chomsky

N.

Lasnik

H.

in

Syntax

An

International

Handbook

of

Contemporary

Research

-LRB-

ed

Jacobs

J.

-RRB-

-LRB-

de

Gruyte

Berlin

-RRB-

Dougherty

R.

C.

Natural

Language

Computing

-LRB-

Lawrence

Erlbaum

Hillsdale

-RRB-

Stabler

E.

P.

Keenan

E.

L.

Structural

similarity

within

and

among

languages

Theor

Comput

Sci

-LRB-

-RRB-

Niyogi

P

The

Computational

Nature

of

Language

Learning

and

Evolution

-LRB-

MIT

Press

Cambridge

-RRB-

Labov

W.

Transmission

and

diffusion

Language

-LRB-

-RRB-

Pinker

S.

Words

and

Rules

The

Ingredients

of

Language

-LRB-

Basic

Books

New

York

-RRB-

Kroch

A.

Reflexes

of

grammar

in

patterns

of

language

change

Lang

Var

Change

-LRB-

-RRB-

Kroch

A.

in

Papers

from

the

Regional

Meeting

of

the

Chicago

Linguistics

Society

Parasession

on

Variation

and

Linguistic

Theory

-LRB-

eds

Beals

K.

et

al.

-RRB-

-LRB-

CLS

Chicago

-RRB-

Pinker

S

The

irregular

verbs

Landfall

-LRB-

Autumn

issue

-RRB-

Bybee

J.

Morphology

a

Study

of

Relation

Between

Meaning

and

Form

-LRB-

Benjamins

Amsterdam

-RRB-

Greenberg

J.

in

Current

Trends

in

Linguistics

III

-LRB-

eds

Sebeok

T.

A.

et

al.

-RRB-

-LRB-

Mouton

The

Hague

-RRB-

Bybee

J.

From

usage

to

grammar

the

mind

's

response

to

repetition

Language

-LRB-

-RRB-

Corbett

G.

Hippisley

A.

Brown

D.

Marriott

P.

in

Frequency

and

the

Emergence

of

Linguistic

Structure

-LRB-

eds

Bybee

J.

Hopper

P.

-RRB-

-LRB-

Benjamins

Amsterdam

-RRB-

Hare

M.

Elman

J.

Learning

and

morphological

change

Cognition

-LRB-

-RRB-

Marcus

G.

Brinkmann

U.

Clahsen

H.

Wiese

R.

Pinker

S.

German

inflection

the

exception

that

proves

the

rule

Cognit

Psychol

-LRB-

-RRB-

Van

der

Wouden

T.

in

Papers

from

the

International

EURALEX

Congress

-LRB-

eds

Magay

T.

Zigany

J.

-RRB-

-LRB-

Akademiai

Kiado

Budapest

-RRB-

Zipf

G.

K.

Human

Behavior

and

the

Principle

of

Least

Effort

-LRB-

Addison-Wesley

Cambridge

-RRB-

Miller

G.

A

Some

effects

of

intermittent

silence

Am

J.

Psychol

-LRB-

-RRB-

Yang

C.

Knowledge

and

Learning

in

Natural

Language

-LRB-

Oxford

Univ.

Press

New

York

-RRB-

Glushko

M.

Towards

the

quantitative

approach

to

studying

evolution

of

English

verb

paradigm

Proc

Scand

Conf

Ling

-LRB-

-RRB-

Kroch

A.

Taylor

A.

Penn-Helsinki

Parsed

Corpus

of

Middle

English

-LSB-

CD-ROM

-RSB-

edn

-LRB-

-RRB-

Supplementary

Information

is

linked

to

the

online

version

of

the

paper

at

Acknowledgements

This

work

was

supported

by

the

John

Templeton

Foundation

and

by

a

grant

from

the

NSF-NIH

joint

programme

in

mathematical

biology

The

Program

for

Evolutionary

Dynamics

is

sponsored

by

J.

Epstein

E.L.

was

supported

by

the

National

Defense

Science

and

Engineering

Graduate

Fellowship

and

the

National

Science

Foundation

Graduate

Fellowship

We

thank

S.

Pinker

J.

Rau

D.

Donoghue

and

A.

Presser

for

discussions

and

J.

Saragosti

for

help

with

visualization

Author

Information

Reprints

and

permissions

information

is

available

at

The

authors

declare

no

competing

financial

interests

Correspondence

and

requests

for

materials

should

be

addressed

to

E.L.

-LRB-

erez@erez.com

-RRB-

Nature

Publishing

Group

O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O