Biostatistics

-LRB-

-RRB-

pp.

Printed

in

Great

Britain

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

RAFAEL

A.

IRIZARRY

Department

of

Biostatistics

Johns

Hopkins

University

Baltimore

MD

USA

rafa@jhu.edu

BRIDGET

HOBBS

Division

of

Genetics

and

Bioinformatics

WEHI

Melbourne

Australia

FRANCOIS

COLLIN

Gene

Logic

Inc.

Berkeley

CA

USA

YASMIN

D.

BEAZER-BARCLAY

KRISTEN

J.

ANTONELLIS

UWE

SCHERF

Gene

Logic

Inc.

Gaithersburg

MD

USA

TERENCE

P.

SPEED

Division

of

Genetics

and

Bioinformatics

WEHI

Melbourne

Australia

Department

of

Statistics

University

of

California

at

Berkeley

S

UMMARY

In

this

paper

we

report

exploratory

analyses

of

high-density

oligonucleotide

array

data

from

the

Affymetrix

GeneChip

R

system

with

the

objective

of

improving

upon

currently

used

measures

of

gene

expression

Our

analyses

make

use

of

three

data

sets

a

small

experimental

study

consisting

of

five

mouse

GeneChip

R

arrays

part

of

the

data

from

an

extensive

spike-in

study

conducted

by

Gene

Logic

and

Wyeth

's

Genetics

Institute

involving

human

GeneChip

R

arrays

and

part

of

a

dilution

study

conducted

by

Gene

Logic

involving

GeneChip

R

arrays

We

display

some

familiar

features

of

the

perfect

match

and

mismatch

probe

-LRB-

P

M

and

M

M

-RRB-

values

of

these

data

and

examine

the

variance

mean

relationship

with

probe-level

data

from

probes

believed

to

be

defective

and

so

delivering

noise

only

We

explain

why

we

need

to

normalize

the

arrays

to

one

another

using

probe

level

intensities

We

then

examine

the

behavior

of

the

P

M

and

M

M

using

spike-in

data

and

assess

three

commonly

used

summary

measures

Affymetrix

's

-LRB-

i

-RRB-

average

difference

-LRB-

AvDiff

-RRB-

and

-LRB-

ii

-RRB-

MAS

signal

and

-LRB-

iii

-RRB-

the

Li

and

Wong

multiplicative

model-based

expression

index

-LRB-

MBEI

-RRB-

The

exploratory

data

analyses

of

the

probe

level

data

motivate

a

new

summary

measure

that

is

a

robust

multiarray

average

-LRB-

RMA

-RRB-

of

background-adjusted

normalized

and

log-transformed

P

M

values

We

evaluate

the

four

expression

summary

measures

using

the

dilution

study

data

assessing

their

behavior

in

terms

of

bias

variance

and

-LRB-

for

MBEI

and

RMA

-RRB-

model

fit

Finally

we

evaluate

the

algorithms

in

terms

of

their

ability

to

detect

known

levels

of

differential

expression

using

the

spike-in

data

We

conclude

that

there

is

no

obvious

downside

to

using

RMA

and

attaching

a

standard

error

-LRB-

SE

-RRB-

to

this

quantity

using

a

linear

model

which

removes

probe-specific

affinities

To

whom

correspondence

should

be

addressed

c

Oxford

University

Press

-LRB-

-RRB-

R.

A.

I

RIZARRY

ET

AL.

An

R

package

with

the

functions

used

for

the

analyses

in

this

paper

is

part

of

the

Bioconductor

project

and

can

be

downloaded

-LRB-

-RRB-

Supplemental

material

such

as

color

versions

of

the

figures

is

available

on

the

web

-LRB-

-RRB-

I

NTRODUCTION

High-density

oligonucleotide

expression

array

technology

is

now

widely

used

in

many

areas

of

biomedical

research

The

system

-LRB-

Lockhart

et

al.

-RRB-

uses

oligonucleotides

with

length

of

base

pairs

that

are

used

to

probe

genes

Typically

each

gene

will

be

represented

by

pairs

of

oligonucleotides

referred

to

as

probe

sets

The

first

component

of

these

pairs

is

referred

to

as

a

perfect

match

-LRB-

P

M

-RRB-

probe

Each

P

M

probe

is

paired

with

a

mismatch

-LRB-

M

M

-RRB-

probe

that

is

created

by

changing

the

middle

-LRB-

-RRB-

base

with

the

intention

of

measuring

non-specific

binding

The

P

M

and

M

M

are

referred

to

as

a

probe

pair

See

the

Affymetrix

Microarray

Suite

User

Guide

-LRB-

-RRB-

for

details

RNA

samples

are

prepared

labeled

and

hybridized

with

arrays

Arrays

are

scanned

and

images

are

produced

and

analysed

to

obtain

an

intensity

value

for

each

probe

These

intensities

represent

how

much

hybridization

occurred

for

each

oligonucleotide

probe

Of

interest

is

finding

a

way

to

combine

the

probe

pair

intensities

for

a

given

gene

to

define

a

measure

of

expression

that

represents

the

amount

of

the

corresponding

mRNA

species

We

denote

the

intensities

obtained

for

each

probe

as

P

Mi

jn

and

M

Mi

jn

i

I

j

Jn

and

n

N

with

n

representing

the

different

genes

i

representing

different

RNA

samples

and

j

representing

the

probe

pair

number

-LRB-

this

number

is

related

to

the

physical

position

of

the

oligonucleotide

in

the

gene

-RRB-

The

number

of

genes

N

usually

ranges

from

to

the

number

of

arrays

I

ranges

from

one

to

hundreds

and

the

number

of

probe

pairs

within

each

gene

Jn

usually

ranges

from

to

Throughout

the

text

indices

are

suppressed

when

there

is

no

ambiguity

Section

describes

the

three

data

sets

used

in

this

paper

Section

explores

various

interesting

features

of

the

data

with

the

objective

of

defining

an

effective

measure

of

gene

expression

using

the

probe

level

data

Section

describes

normalization

Some

expression

measures

for

example

AvDiff

and

Li

and

Wong

's

MBEI

are

based

on

P

M

M

M

Other

measures

for

example

Affymetrix

's

Average

Log

Ratio

are

based

on

log

-LRB-

P

M/M

M

-RRB-

In

Sections

and

we

also

explore

the

behavior

of

these

quantities

Section

describes

four

measures

of

expression

Section

assesses

the

four

expression

measures

in

terms

of

bias

variance

and

model

fit

Section

examines

the

ability

of

the

four

methods

at

detecting

differentially

expressed

probe

sets

Section

presents

our

conclusions

D

ESCRIPTION

OF

DATA

To

properly

compare

summary

measures

of

expression

in

terms

of

bias

variance

sensitivity

and

specificity

data

for

which

we

know

the

truth

is

required

In

this

paper

we

examine

three

data

sets

for

which

assessments

can

be

performed

where

specific

results

are

expected

Data

set

A

provides

probes

for

which

we

can

assume

the

measurements

are

entirely

due

to

non-specific

binding

This

permits

us

to

study

the

variance

mean

relationship

for

intensity

measures

Data

set

B

provides

the

results

of

a

spike-in

experiment

where

gene

fragments

have

been

added

at

known

concentrations

These

data

can

be

used

to

assess

bias

sensitivity

and

specificity

Data

set

C

provides

the

results

from

a

study

in

which

samples

were

hybridized

at

different

dilutions

This

permits

us

to

assess

bias

and

variance

in

a

more

realistic

scenario

than

with

data

set

B.

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

Data

sets

B

and

C

are

available

from

the

web

at

In

this

section

we

describe

them

in

detail

for

readers

interested

in

using

them

We

also

explain

which

specific

subsets

of

the

data

were

used

for

the

analyses

presented

in

this

paper

Mouse

data

set

A

Data

set

A

comes

from

an

experiment

where

five

mouse

GeneChip

R

arrays

were

used

These

were

hybridized

with

samples

of

lung

tissue

mRNA

obtained

from

five

mice

exposed

to

different

experimental

conditions

About

of

the

probe

pairs

in

the

array

were

incorrectly

sequenced

We

therefore

assume

that

the

measurements

read

for

most

of

these

probes

are

entirely

due

to

non-specific

binding

Spike-in

data

sets

B

Data

set

B

consists

of

experiments

where

different

cRNA

fragments

were

added

to

the

hybridization

mixture

of

the

GeneChip

R

arrays

at

different

picomolar

-LRB-

pM

-RRB-

concentrations

The

control

cRNAs

were

BioB-M

-LRB-

all

E.

coli

-RRB-

-LRB-

phage

-RRB-

and

DapX-M

-LRB-

B.

subtilis

-RRB-

-LRB-

Hill

et

al.

Baugh

et

al.

-RRB-

The

cRNA

were

chosen

to

match

the

target

sequence

for

each

of

the

Affymetrix

control

probe

sets

For

example

for

DapX

-LRB-

a

B.

subtilis

gene

-RRB-

the

middle

and

target

sequences

-LRB-

identified

by

DapX-M

-RRB-

were

each

synthesized

separately

and

spiked-in

at

a

specific

concentration

Thus

for

example

on

one

of

the

arrays

target

sequence

was

added

to

the

total

hybridization

solution

of

l

to

give

a

final

concentration

of

pM

There

are

two

series

of

spike-in

experiments

The

experiments

were

originally

carried

out

for

the

development

of

normalization

procedures

-LRB-

Hill

et

al.

-RRB-

In

this

paper

we

use

the

data

in

a

different

way

mainly

for

the

comparison

of

expression

measures

The

varying

concentration

series

data

set

For

an

individual

array

all

of

the

control

cRNAs

were

spiked-in

at

the

same

concentration

and

this

concentration

was

varied

across

arrays

taking

the

values

and

pM

For

example

array

had

all

control

cRNAs

spiked

with

pM

and

array

had

all

control

cRNAs

spiked

with

pM

etc.

Of

these

concentrations

were

represented

on

just

one

array

and

on

two

arrays

and

the

rest

were

in

triplicate

i.e.

on

three

arrays

for

a

total

of

arrays

All

arrays

have

a

common

background

cRNA

from

an

acute

myeloid

leukemia

-LRB-

AML

-RRB-

tumor

cell

line

In

this

paper

we

use

only

arrays

one

replicate

for

each

of

the

concentrations

One

of

the

probe

set

spike-in

combinations

-LRB-

-RRB-

failed

to

respond

adequately

and

data

from

that

probe

set

is

entirely

omitted

from

the

analysis

Thus

we

analyse

data

from

spiked-in

probe-sets

Latin

square

series

data

set

In

this

series

each

of

the

control

cRNAs

were

spiked-in

at

a

different

concentration

on

each

array

-LRB-

apart

from

replicates

-RRB-

The

concentrations

used

were

and

pM

and

these

were

arranged

in

a

cyclic

Latin

square

with

each

concentration

appearing

once

in

each

row

and

column

The

combinations

of

concentrations

used

on

the

arrays

were

taken

from

the

first

entries

of

the

rows

of

this

Latin

square

Of

the

combinations

used

were

done

on

three

arrays

and

one

on

just

one

array

All

of

these

arrays

had

the

same

AML

background

as

in

data

set

The

analysis

in

this

paper

makes

use

of

data

from

six

arrays

that

are

a

pair

of

triplicates

The

spikein

concentrations

for

each

of

the

control

RNAs

on

the

two

sets

of

triplicates

is

shown

in

Table

R.

A.

I

RIZARRY

ET

AL.

Table

Concentrations

and

observed

ranks

of

each

spiked-in

gene

in

a

comparison

of

two

sets

of

triplicates

from

the

Latin

square

series

spike-in

data

set

Probe

set

BioB-M

DapX-M

Concentration

Set

of

triplicates

set

of

triplicates

Expected

Rank

AvDiff

30 44 239 333

3276 2709 4598 165

Observed

Rank

MAS

Li

Wong

36 33 3128 8579

681 102 12203

RMA

Notice

that

relative

concentrations

of

the

spike-ins

are

three

fold

or

more

which

permits

us

to

check

the

sensitivity

of

expression

indices

Dilution

data

set

C

Two

sources

of

cRNA

A

-LRB-

human

liver

tissue

-RRB-

and

B

-LRB-

central

nervous

system

cell

line

-RRB-

were

hybridized

to

human

array

-LRB-

-RRB-

in

a

range

of

proportions

and

dilutions

In

this

publication

we

study

data

from

arrays

hybridized

to

source

A

starting

with

g

cRNA

and

rising

through

to

g

There

were

five

replicate

arrays

for

each

tissue

that

is

each

generated

cRNA

was

hybridized

on

five

GeneChip

R

arrays

Five

scanners

were

used

in

this

study

Each

array

replicate

was

processed

in

a

different

scanner

F

EATURES

OF

PROBE

LEVEL

DATA

Figure

-LRB-

a

-RRB-

shows

histograms

of

log

ratio

-LRB-

P

M/M

M

-RRB-

stratified

by

quantiles

of

abundance

P

M

M

M

with

gray

scale

representing

height

of

histogram

-LRB-

light

is

high

and

dark

is

low

-RRB-

for

one

array

from

data

set

A

The

histograms

have

been

scaled

so

that

the

mode

of

each

histogram

is

represented

with

the

same

gray

scale

This

figure

shows

that

in

general

M

M

grows

with

P

M.

Furthermore

for

larger

values

of

abundance

the

differences

have

a

bimodal

distribution

with

the

second

mode

occurring

for

negative

differences

The

same

bimodal

effect

is

seen

when

we

stratify

by

-LRB-

P

M

-RRB-

thus

it

is

not

an

artifact

of

conditioning

on

sums

In

Figures

-LRB-

b

-RRB-

-LRB-

e

-RRB-

four

histograms

with

a

broader

stratification

clearly

show

this

effect

The

figure

also

displays

-LRB-

in

darker

grays

-RRB-

the

histograms

of

the

defective

probes

where

the

bimodal

distribution

is

also

seen

Notice

there

are

many

probe

pairs

with

M

M

P

M.

Finally

notice

that

for

about

of

the

probes

M

M

P

M

The

number

of

probe

pairs

within

probe

sets

for

which

M

M

P

M

varies

from

to

The

distribution

across

probe

sets

is

the

following

of

times

M

M

P

M

of

probe

sets

7401 481 628 819

1123 1461 1759 1906

1555 1200 760 345

All

these

effects

have

been

seen

in

many

arrays

The

defective

probes

are

used

to

assess

the

variance

mean

relationship

Intensities

obtained

from

probe

j

in

arrays

i

I

P

Mi

jn

are

expected

to

have

the

same

mean

and

variance

If

standard

deviations

-LRB-

SDs

-RRB-

-LCB-

-LRB-

I

-RRB-

-LRB-

P

Mi

jn

P

jn

-RRB-

-RCB-

and

averages

P

jn

I

i

P

Mi

jn

are

computed

M

M

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

-LRB-

a

-RRB-

-LRB-

pm/mm

-RRB-

Histograms

by

-LRB-

pmxmm

-RRB-

-LRB-

pm/mm

-RRB-

-LRB-

pmxmm

-RRB-

quantile

-LRB-

b

-RRB-

low

-LRB-

-RRB-

abundance

-LRB-

c

-RRB-

medium

-LRB-

-RRB-

abundance

-LRB-

d

-RRB-

high

-LRB-

-RRB-

abundance

-LRB-

e

-RRB-

very

-LRB-

-RRB-

high

abundance

200 400 600

Fig.

-LRB-

a

-RRB-

Histograms

of

log

ratio

-LRB-

P

M/M

M

-RRB-

stratified

by

quantiles

of

abundance

P

M

M

M

with

gray

scale

representing

height

of

histogram

-LRB-

light

grays

are

high

and

dark

grays

are

low

-RRB-

for

one

array

from

the

mouse

data

set

The

histograms

have

been

scaled

so

that

the

mode

of

each

histogram

is

represented

with

the

same

gray

scale

-LRB-

b

-RRB-

Histogram

of

log

ratios

for

first

quartile

of

abundance

with

the

histogram

for

the

defective

probes

represented

by

a

darker

gray

-LRB-

c

-RRB-

Like

-LRB-

b

-RRB-

for

abundance

values

between

first

and

third

quartile

-LRB-

d

-RRB-

Like

-LRB-

b

-RRB-

for

abundance

values

in

the

last

quartile

excluding

the

highest

percent

-LRB-

e

-RRB-

Like

-LRB-

b

-RRB-

for

the

highest

of

abundance

R.

A.

I

RIZARRY

ET

AL.

for

a

random

sample

of

defective

probe

sets

the

SD

increases

from

roughly

to

a

factor

of

as

the

average

increases

on

its

entire

range

After

a

log

transformation

of

the

P

M

intensities

there

is

only

a

increase

N

ORMALIZATION

In

many

of

the

applications

of

high-density

oligonucleotide

arrays

the

goal

is

to

learn

how

RNA

populations

differ

in

expression

in

response

to

genetic

and

environmental

differences

For

example

large

expression

of

a

particular

gene

or

genes

may

cause

an

illness

resulting

in

variation

between

diseased

and

normal

tissue

These

sources

of

variation

are

referred

to

as

interesting

variation

Observed

expression

levels

also

include

variation

introduced

during

the

sample

preparation

manufacture

of

the

arrays

and

the

processing

of

the

arrays

-LRB-

labeling

hybridization

and

scanning

-RRB-

These

are

referred

to

as

sources

of

obscuring

variation

See

-LRB-

Hartemink

et

al.

-RRB-

for

a

more

detailed

discussion

The

obscuring

sources

of

variation

can

have

many

different

effects

on

data

Unless

arrays

are

appropriatly

normalized

comparing

data

from

different

arrays

can

lead

to

misleading

results

Dudoit

et

al.

-LRB-

-RRB-

describe

the

need

for

normalization

procedures

for

cDNA

microarray

data

Similar

issues

are

present

with

GeneChip

R

arrays

Figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

show

box

plots

of

-LRB-

P

M

-RRB-

and

P

M

M

M

for

data

set

C

The

different

gray

scales

represent

the

six

different

sets

of

five

replicates

processed

on

scanners

to

The

scanner

effect

is

clearly

seen

in

Figure

For

example

note

that

the

-LRB-

P

M

-RRB-

boxplot

intensities

obtained

using

scanner/fluidic

station

were

in

general

higher

than

those

obtained

from

scanner/fluidic

station

For

the

replicate

arrays

we

expect

no

genes

to

be

differentially

expressed

This

figure

shows

direct

array

to

array

comparison

of

P

M

values

warrants

normalization

Figure

-LRB-

b

-RRB-

boxplot

shows

that

further

normalization

is

needed

for

the

P

M

M

M

as

well

Figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

show

log

ratios

M

-LRB-

y/x

-RRB-

versus

abundance

A

x

y

-LRB-

MVA

-RRB-

plots

for

x

P

y

P

and

x

P

M

y

P

M

for

two

arrays

-LRB-

denoted

with

and

-RRB-

in

which

the

gene

has

been

spiked

at

pM

and

pM

respectively

These

plots

have

been

used

by

for

example

Dudoit

et

al.

-LRB-

-RRB-

to

explore

intensity

related

biases

Because

the

same

RNA

background

was

hybridized

to

arrays

and

we

do

not

expect

any

of

the

non-spiked-in

genes

to

be

differentially

expressed

and

therefore

these

plots

to

scatter

around

It

is

clear

from

Figure

that

these

data

need

normalization

For

cDNA

arrays

the

normalization

procedure

presented

in

Dudoit

et

al.

-LRB-

-RRB-

has

worked

well

in

practice

For

each

array

a

loess

curve

is

fitted

to

the

MVA

plot

of

intensities

of

the

red

and

green

labels

and

the

residuals

are

considered

the

normalized

log

ratios

However

this

approach

is

not

appropriate

for

GeneChip

R

arrays

because

only

one

sample

is

hybridized

to

each

array

instead

of

two

-LRB-

red

and

green

-RRB-

A

procedure

that

normalizes

each

array

against

all

others

is

needed

Various

methods

have

been

proposed

for

normalizing

GeneChip

R

arrays

Bolstad

et

al.

-LRB-

-RRB-

present

a

review

of

these

methods

and

find

quantile

normalization

to

perform

best

The

goal

of

quantile

normalization

is

to

make

the

distribution

of

probe

intensities

the

same

for

arrays

i

I

The

normalization

maps

probe

level

data

from

all

arrays

i

I

so

that

an

I

dimensional

quantile

quantile

plot

follows

the

I

dimensional

identity

line

A

possible

problem

with

this

approach

is

that

we

risk

removing

some

of

the

signal

in

the

tails

However

empirical

evidence

suggest

this

is

not

a

problem

in

practice

see

Bolstad

et

al.

-LRB-

-RRB-

for

details

In

Figures

-LRB-

c

-RRB-

and

-LRB-

d

-RRB-

the

MVA

plots

of

the

normalized

arrays

are

shown

Notice

how

the

normalization

has

removed

the

bias

seen

in

Figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

The

large

points

represent

the

spiked-in

probes

and

the

small

black

dots

represent

a

random

sample

of

non-spiked-in

probes

Notice

that

in

all

plots

normalization

helps

identify

the

spiked-in

probes

as

differentially

expressed

The

benefits

of

this

normalization

at

the

probe

level

are

also

seen

in

Figures

-LRB-

c

-RRB-

and

-LRB-

d

-RRB-

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

d

-RRB-

PM-MM

data

after

normalization

b

-RRB-

Raw

PM-MM

data

Concentrations

Concentrations

50 100 150 200

Concentrations

Fig.

Boxplots

of

-LRB-

P

M

-RRB-

and

P

M

M

M

for

the

arrays

from

data

set

C.

Because

P

M

M

M

values

are

usually

between

and

a

reduced

range

is

used

to

get

a

better

view

of

the

interquartile

range

The

bottom

row

are

the

after

quantile

normalization

boxplots

The

y-axis

scale

can

be

deduced

from

the

plot

titles

M

EASURES

OF

EXPRESSION

Various

measures

of

expression

have

been

proposed

for

example

see

Li

and

Wong

-LRB-

-RRB-

Naef

et

al.

-LRB-

-RRB-

and

Holder

et

al.

-LRB-

-RRB-

The

most

commonly

used

-LRB-

at

the

time

this

paper

was

written

-RRB-

is

AvDiff

c

-RRB-

Normalized

PM

data

a

-RRB-

Raw

PM

data

Concentrations

R.

A.

I

RIZARRY

ET

AL.

c

-RRB-

PM-MM

before

normalization

d

-RRB-

PM-MM

after

normalization

A

A

M

M

A

a

-RRB-

log

-LRB-

PM

-RRB-

before

normalization

Fig.

MVA

plots

-LRB-

described

in

text

-RRB-

of

-LRB-

P

M

-RRB-

and

-LRB-

P

M

M

M

-RRB-

for

two

arrays

in

which

the

gene

has

been

spiked

at

pM

and

pM

respectively

The

large

points

represent

the

spiked-in

probes

and

the

small

black

dots

represent

a

random

sample

of

non-spiked-in

probes

-LRB-

a

-RRB-

and

-LRB-

c

-RRB-

are

before

normalization

and

-LRB-

b

-RRB-

and

-LRB-

d

-RRB-

are

after

quantile

normalization

the

Affymetrix

default

For

each

probe

set

n

on

each

array

i

AvDiff

is

defined

by

AvDiff

A

-LRB-

P

M

j

M

M

j

-RRB-

jA

b

-RRB-

log

-LRB-

PM

-RRB-

after

normalization

A

M

M

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

with

A

the

subset

of

probes

for

which

d

j

P

M

j

M

M

j

are

within

SDs

away

from

the

average

of

d

-LRB-

-RRB-

d

-LRB-

J

-RRB-

with

d

-LRB-

j

-RRB-

the

jth

smallest

difference

A

represents

the

cardinality

of

A

Many

of

the

other

expression

measures

are

versions

of

AvDiff

with

different

ways

of

removing

outliers

and

different

ways

of

dealing

with

small

values

We

have

observed

that

linear

scale

measures

such

as

AvDiff

are

not

optimal

Li

and

Wong

-LRB-

-RRB-

observed

this

and

proposed

an

alternative

model

based

expression

index

For

each

probe

set

n

Li

and

Wong

's

measure

is

defined

as

the

maximum

likelihood

estimates

of

the

i

i

I

obtained

from

fitting

P

Mi

j

M

Mi

j

i

j

ij

-LRB-

-RRB-

with

j

representing

probe-specific

affinities

and

the

i

jn

are

assumed

to

be

independent

normally

distributed

errors

The

estimation

procedure

includes

rules

for

outlier

removal

Affymetrix

also

appears

to

have

noticed

that

the

linear

scale

is

not

appropriate

and

in

the

new

version

of

their

analysis

algorithm

MAS

are

now

using

a

log

scale

measure

Specifically

the

MAS

signal

-LRB-

measure

-RRB-

is

defined

as

signal

Tukey

Biweight

-LCB-

log

-LRB-

P

M

j

C

T

j

-RRB-

-RCB-

with

C

T

j

a

quantity

derived

from

the

M

Ms

that

is

never

bigger

than

its

P

M

pair

See

Hubbell

-LRB-

-RRB-

for

more

details

Each

of

these

measures

rely

upon

the

difference

P

M

M

M

with

the

intention

of

correcting

for

nonspecific

binding

However

the

exploratory

analysis

presented

in

Section

suggests

that

the

M

M

may

be

detecting

signal

as

well

as

non-specific

binding

Some

researchers

-LRB-

Naef

et

al.

-RRB-

propose

expression

measures

based

only

on

the

P

M

In

Figure

we

show

the

P

M

M

M

P

M/M

M

and

P

M

M

M

values

for

each

of

the

probes

representing

in

the

spiked-in

arrays

from

data

set

plotted

against

spike-in

concentration

The

different

probe

pairs

are

represented

with

different

symbols

and

line

types

As

expected

the

P

M

values

are

growing

in

proportion

to

the

concentration

Notice

also

that

the

lines

representing

the

probes

are

close

to

being

parallel

showing

there

is

a

strong

additive

-LRB-

in

the

log

scale

-RRB-

probe-specific

effect

As

evident

in

Figure

-LRB-

c

-RRB-

the

additive

probe-specific

effect

is

also

detected

by

the

M

M

motivating

their

subtraction

from

the

P

M.

However

in

Figure

-LRB-

d

-RRB-

the

parallel

lines

are

still

seen

in

P

M

M

M

demonstrating

that

subtracting

is

not

enough

to

remove

the

probe

effect

The

fact

that

parallel

lines

are

not

as

obvious

in

Figure

-LRB-

c

-RRB-

shows

that

dividing

by

M

M

removes

to

some

degree

the

probe

effect

However

the

M

M

also

grow

with

concentrations

because

they

detect

signal

as

well

as

non-specific

binding

hence

the

signal

in

P

M/M

M

is

attenuated

Notice

in

particular

that

P

M/M

M

is

unable

to

distinguish

between

concentrations

of

and

Since

subtracting

probe-specific

M

M

adds

noise

with

no

obvious

gain

in

bias

and

because

P

M/M

M

results

in

a

biased

signal

in

this

paper

we

propose

an

alternative

measure

to

those

based

on

P

M

M

M

or

P

M/M

M.

Figure

-LRB-

a

-RRB-

shows

that

on

a

log

scale

-LRB-

i

-RRB-

the

P

Ms

grow

roughly

linearly

with

respect

to

concentrations

-LRB-

ii

-RRB-

the

variances

are

roughly

constant

and

-LRB-

iii

-RRB-

the

probe-specific

affinity

is

approximately

additive

This

suggests

an

additive

linear

model

for

the

probe

set

data

and

the

average

J

J

log

-LRB-

P

Mi

j

-RRB-

as

a

log

j

scale

measure

of

expression

However

this

measure

does

not

account

for

non-specific

binding

Because

in

Figure

the

log-scale

slope

of

the

P

M

is

less

than

particularly

for

small

concentrations

the

P

M

values

should

be

adjusted

to

account

for

non-specific

binding

To

see

this

consider

a

hypothetical

case

with

two

arrays

where

the

signal

of

a

probe

set

is

twice

as

big

in

one

of

the

arrays

but

an

additive

signal

of

units

occurs

due

to

non-specific

binding

and/or

background

noise

in

both

arrays

In

this

case

the

observed

difference

in

the

signals

would

be

about

-LRB-

-RRB-

-LRB-

s

-RRB-

instead

of

-LRB-

-RRB-

-LRB-

s

-RRB-

For

small

values

of

s

the

incorrect

difference

would

be

close

to

instead

of

Figure

shows

histograms

of

-LRB-

M

M

-RRB-

for

an

array

in

which

no

probe-set

was

spiked

along

with

the

three

arrays

in

which

was

spiked-in

at

concentrations

of

and

pM

-LRB-

from

data

set

R.

A.

I

RIZARRY

ET

AL.

d

-RRB-

PM-MM

concentration

800 600 400 200

concentration

5000 2000 1000 500

MM

concentration

5000 2000 1000 500

Fig.

P

M

M

M

P

M/M

M

and

P

M

M

M

values

for

each

of

the

probes

representing

-LRB-

with

the

exception

of

all

other

spike-in

genes

behaved

similarly

to

-RRB-

in

the

spiked-in

arrays

from

the

varying

concentration

experiment

plotted

against

concentration

The

different

probes

are

represented

by

the

different

line

types

and

symbols

The

horizontal

line

represents

the

median

of

the

probes

for

the

non-spiked-in

array

The

dashed

lines

are

the

and

quantiles

b

-RRB-

MM

a

-RRB-

PM

PM-MM

PM/MM

PM

concentration

c

-RRB-

PM/MM

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

c

-RRB-

concentration

of

d

-RRB-

concentration

of

MM

b

-RRB-

concentration

of

a

-RRB-

concentration

of

MM

Density

50 100 150

MM

Fig.

Histograms

of

-LRB-

M

M

-RRB-

for

a

array

in

which

no

probe-set

was

spiked

along

with

the

three

arrays

in

which

was

spiked-in

at

concentrations

of

and

pM

The

observed

P

M

values

for

the

probes

associated

with

are

marked

with

crosses

and

the

average

with

an

arrow

The

black

curve

represents

the

log

normal

distribution

obtained

from

left-of-the-mode

data

Density

MM

Density

Density

R.

A.

I

RIZARRY

ET

AL.

-RRB-

All

arrays

in

all

data

sets

had

similar

shaped

-LRB-

M

M

-RRB-

histograms

Furthermore

the

-LRB-

M

M

-RRB-

histograms

for

the

spiked-in

probe

set

had

similar

histograms

as

well

The

M

Ms

to

the

left

of

the

mode

of

the

histogram

can

be

approximated

with

the

left-hand

tail

of

a

log-normal

distribution

This

suggests

that

the

M

Ms

are

a

mixture

of

probes

for

which

-LRB-

i

-RRB-

the

intensities

are

largely

due

to

non-specific

binding

and

background

noise

and

-LRB-

ii

-RRB-

the

intensities

include

transcript

signal

just

like

the

P

Ms.

The

mode

of

the

histogram

is

a

natural

estimate

of

the

mean

background

level

The

observed

P

M

values

for

the

probes

associated

with

are

marked

with

crosses

and

the

average

with

an

arrow

All

the

average

P

M

values

are

close

to

Thus

judging

solely

on

the

average

a

difference

would

be

hard

to

detect

However

distance

of

the

average

P

M

from

the

average

background

noise

does

in

fact

increase

with

concentration

Figure

motivates

a

background

plus

signal

model

of

the

form

P

Mi

jn

bgi

jn

si

jn

Here

bgi

jn

represents

background

signal

in

array

i

caused

by

optical

noise

and

non-specific

binding

We

assume

each

array

has

a

common

mean

background

level

E

-LRB-

bgi

jn

-RRB-

i

We

want

to

adjust

the

P

M

intensities

to

remove

the

background

effect

A

naive

approach

is

to

consider

P

Mi

jn

i

with

-LRB-

i

-RRB-

the

mode

of

the

-LRB-

M

M

-RRB-

distribution

An

estimate

of

this

distribution

can

be

obtained

using

a

density

kernel

i

estimate

In

practice

a

problem

with

this

measure

is

that

for

a

small

percentage

of

probes

P

Mi

jn

and

log

transforming

P

Mi

jn

i

becomes

a

problem

An

alternative

background

correction

is

to

consider

B

-LRB-

P

Mi

jn

-RRB-

E

-LRB-

si

jn

P

Mi

jn

-RRB-

If

we

impose

a

strictly

positive

distribution

on

si

jn

then

B

-LRB-

P

Mi

jn

-RRB-

To

obtain

a

computationally

feasible

B

-LRB-

-RRB-

we

consider

the

closed-form

transformation

obtained

when

assuming

si

jn

is

exponential

and

bgi

jn

is

normal

Although

the

data

suggest

that

this

model

can

be

improved

the

results

obtained

using

B

-LRB-

-RRB-

work

well

in

practice

as

is

demonstrated

in

the

next

section

To

obtain

an

expression

measure

we

assume

that

for

each

probe

set

n

the

background-adjusted

normalized

and

log-transformed

P

M

intensities

denoted

with

Y

follow

a

linear

additive

model

Yi

jn

in

jn

i

jn

i

I

j

J

n

n

-LRB-

-RRB-

with

j

a

probe

affinity

effect

i

representing

the

log

scale

expression

level

for

array

i

and

i

j

representing

an

independent

identically

distributed

error

term

with

mean

For

identifiability

of

the

parameters

we

assume

that

j

j

for

all

probe

sets

This

assumption

is

saying

that

Affymetrix

technology

has

chosen

probes

with

intensities

that

on

average

are

representative

of

the

associated

genes

expression

The

estimate

of

i

gives

the

expression

measures

for

probe

set

n

on

array

i.

To

summarize

in

this

paper

we

consider

a

new

expression

measure

that

-LRB-

i

-RRB-

background-corrects

the

arrays

using

the

transformation

B

-LRB-

-RRB-

-LRB-

ii

-RRB-

normalizes

the

arrays

using

quantile

normalization

and

-LRB-

iii

-RRB-

for

each

probe

set

n

fits

a

linear

model

-LRB-

-RRB-

to

the

background-corrected

normalized

and

log

-LRB-

base

-RRB-

transformed

probe

intensities

denoted

here

with

Yi

j

i

I

j

J

To

protect

against

outlier

probes

we

use

a

robust

procedure

such

as

median

polish

-LRB-

Holder

et

al.

-RRB-

to

estimate

model

parameters

We

use

the

estimate

of

i

as

the

log

scale

measure

of

expression

which

we

refer

to

as

robust

multi-array

average

-LRB-

RMA

-RRB-

B

IAS

VARIANCE

AND

GOODNESS

OF

FIT

COMPARISONS

Plots

of

log

observed

expression

versus

known

concentration

-LRB-

not

shown

-RRB-

demonstrate

that

the

expression

measures

perform

similarly

in

detecting

the

spiked-in

probe

sets

However

for

the

highest

concentration

AvDiff

and

MBEI

sometimes

underestimate

the

predicted

value

from

the

known

concentrations

This

results

from

the

attenuation

caused

by

subtracting

M

M

We

also

notice

that

RMA

is

less

noisy

than

all

other

measures

at

lower

concentrations

It

is

possible

that

the

control

genes

used

in

data

set

provide

a

stronger

than

usual

signal

Therefore

a

comparison

based

on

all

probe

sets

of

the

arrays

is

conducted

using

data

set

C.

For

these

data

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

a

-RRB-

Expression

10 100 1000

AvDiff

MAS

concentrations

MBEI

RMA

b

-RRB-

Standard

deviation

vs.

average

expression

Standard

Deviation

between

Replicates

AvDiff

MAS

MBEI

RMA

Expression

Fig.

Data

set

C

boxplots

-LRB-

a

-RRB-

Averages

over

replicates

for

each

gene

in

-LRB-

b

-RRB-

-LRB-

b

-RRB-

Loess

curves

fitted

to

standard

deviation

versus

average

expression

scatter-plots

the

amount

of

hybridization

of

probe

sets

representing

expressed

genes

is

expected

to

double

when

the

amount

of

RNA

hybridized

to

the

array

is

double

Furthermore

the

difference

in

gene

expression

across

replicate

arrays

should

be

small

For

each

of

the

four

measures

we

denote

the

expression

values

with

E

ik

i

k

with

i

representing

the

dilution

concentration

level

and

k

the

replicate

-LRB-

which

also

identifies

scanner

-RRB-

The

averages

are

denoted

with

E

i

-LRB-

-RRB-

E

ik

and

the

SDs

with

S

Di

-LRB-

-RRB-

-LRB-

E

ik

E

i

-RRB-

k

k

Figure

-LRB-

a

-RRB-

shows

boxplots

of

the

E

i

for

each

dilution

concentration

i.

Notice

that

all

measures

have

R.

A.

I

RIZARRY

ET

AL.

roughly

the

same

ability

to

detect

signal

Figure

-LRB-

b

-RRB-

shows

loess

curves

fitted

to

the

scatter

plot

-LRB-

on

the

log

scale

-RRB-

of

S

Di

vs

E

i

Clearly

RMA

has

the

smallest

SD

across

replicates

The

advantage

of

RMA

is

especially

noticeable

in

the

low

expression

values

where

the

SD

is

times

smaller

than

the

other

measures

Li

and

Wong

's

method

provides

not

only

an

estimate

of

i

but

a

nominal

SE

for

this

estimate

denoted

here

with

i

Under

-LRB-

-RRB-

one

can

obtain

a

naive

nominal

estimate

for

the

SE

of

using

an

analysis

of

variance

approach

Because

there

are

five

replicates

one

can

also

obtain

an

observed

SE

of

any

estimate

by

simply

considering

the

S

Di

defined

above

If

the

model

is

close

to

the

actual

mechanism

giving

rise

to

the

data

the

nominal

and

observed

SE

should

agree

Plots

of

nominal

to

observed

SE

log

ratios

versus

expression

-LRB-

not

shown

-RRB-

show

that

in

general

RMA

is

closer

to

than

Li

and

Wong

's

MBEI

showing

that

the

observed

and

nominal

standard

error

methods

are

in

general

closer

when

using

-LRB-

-RRB-

instead

of

-LRB-

-RRB-

D

ETECTION

OF

DIFFERENTIAL

EXPRESSION

Data

set

was

used

to

assess

how

well

the

different

expression

measures

perform

at

detecting

differentially

expressed

probe

sets

For

each

of

the

six

arrays

studied

expression

measures

E

E

E

E

E

E

were

obtained

in

their

respective

scale

-LRB-

log

for

MAS

and

RMA

-RRB-

for

each

probe

set

n

N

We

then

computed

the

averages

over

triplicates

E

i

n

-LRB-

-RRB-

E

ikn

i

n

N

For

the

probe

sets

representing

spike-in

RNAs

the

observed

k

ratios

or

fold

changes

-LRB-

E

n

E

n

for

AvDiff

and

MBEI

or

E

n

E

n

for

MAS

and

RMA

-RRB-

should

coincide

with

the

true

ratio

of

the

spike-in

concentrations

shown

in

Table

Recall

that

apart

from

the

spiked-in

probe

sets

the

background

samples

hybridized

to

the

six

arrays

are

the

same

We

therefore

expect

only

the

probe

sets

shown

in

Table

to

be

differentially

expressed

In

the

left

side

of

Figure

MVA

plots

of

the

average

expressions

obtained

are

shown

Probe

sets

with

negative

expression

measures

were

left

out

for

AvDiff

and

Li

and

Wong

's

MBEI

Notice

that

all

measures

separate

out

of

the

spiked-in

probe

sets

from

the

cloud

of

points

However

the

cloud

of

points

for

probe

sets

with

small

total

intensity

has

a

much

larger

spread

for

AvDiff

MBEI

and

MAS

than

for

RMA

For

this

reason

many

of

the

probe

sets

with

high

differential

expressions

for

AvDiff

MBEI

and

MAS

are

not

actually

the

spiked-in

probe-sets

The

smaller

spread

of

RMA

results

in

better

detection

of

differentially

expressed

probe-sets

In

the

right

side

of

Figure

quantile

quantile

plots

of

the

observed

ratios

are

shown

RMA

is

the

only

measure

to

perfectly

differentiate

the

spiked-in

probe

sets

-LRB-

with

the

exception

which

no

measure

was

able

to

detect

-RRB-

from

the

rest

Table

shows

the

observed

rank

of

the

spiked-in

probe

sets

C

ONCLUSION

In

this

paper

we

have

developed

a

novel

measure

of

gene

expression

and

compared

it

to

other

standard

measures

Through

the

analyses

of

three

data

sets

we

have

shown

that

expression

is

better

measured

using

log-transformed

PM

values

after

carrying

out

a

global

background

adjustment

and

across-array

normalization

We

studied

the

performance

of

a

version

of

the

Affymetrix

summary

measures

AvDiff

and

MAS

the

Li

and

Wong

model-based

expression

index

and

the

new

measure

RMA

We

evaluated

the

four

expression

summary

measures

using

spike-in

and

dilution

study

data

assessing

their

behavior

in

terms

of

bias

variance

the

ability

to

detect

known

differential

expression

levels

and

-LRB-

for

MBEI

and

RMA

-RRB-

model

fit

We

conclude

that

there

is

no

obvious

downside

to

summarizing

the

expression

level

of

a

probe

set

with

RMA

and

attaching

an

SE

to

this

quantity

using

a

linear

model

that

removes

probespecific

affinities

The

greater

sensitivity

and

specificity

of

RMA

in

detection

of

differential

expression

provides

a

useful

improvement

for

researchers

using

the

GeneChip

R

technology

We

expect

marginal

though

worthwhile

gains

to

be

achievable

by

using

a

more

carefully

designed

and

tested

background

correction

procedure

Exploration

normalization

and

summaries

of

high

density

oligonucleotide

array

probe

level

data

a

-RRB-

AvDiff

MVA

plot

b

-RRB-

AvDiff

QQ-plot

M

observed

quantiles

A

reference

quantiles

c

-RRB-

MAS

MVA

plot

d

-RRB-

MAS

QQ-plot

M

observed

quantiles

A

reference

quantiles

e

-RRB-

Li

and

Wong

's

MVA

plot

f

-RRB-

Li

and

Wong

's

QQ-plot

M

observed

quantiles

A

reference

quantiles

g

-RRB-

RMA

MVA

plot

h

-RRB-

RMA

QQ-plot

M

observed

quantiles

A

reference

quantiles

Fig.

MVA

-LRB-

described

in

text

-RRB-

and

qq-plots

indicating

the

positions

of

differentially

expressed

genes

ranked

by

their

absolute

log

relative

expression

values

R.

A.

I

RIZARRY

ET

AL.

ACKNOWLEDGEMENTS

The

work

of

Rafael

A.

Irizarry

was

supported

by

the

PGA

We

would

like

to

thank

Skip

Garcia

Tom

Cappola

and

Joshua

M.

Hare

from

Johns

Hopkins

University

for

the

mouse

data

and

Gene

Brown

's

group

at

Wyeth/Genetics

Institute

for

helpful

suggestions

in

the

design

of

the

spike-in

experiment

We

would

like

to

thank

Rehannah

Borup

and

Eric

Hoffman

from

the

Children

's

National

Medical

Center

Microarray

Center

for

help

obtaining

the

raw

mouse

data

We

would

like

to

thank

Laurent

Gautier

from

the

Technical

University

of

Denmark

Ben

Bolstad

from

UC

Berkeley

and

Magnus

strand

from

Astra

Zeneca

M

lndal

for

developing

and

coding

up

the

normalization

routines

Finally

we

thank

o

Earl

Hubbell

-LRB-

Affymetrix

-RRB-

Cheng

Li

-LRB-

Harvard

-RRB-

the

Associate

Editor

and

the

referee

for

suggestions

that

have

improved

this

paper

R

EFERENCES

A

FFYMETRIX

-LRB-

-RRB-

Affymetrix

Microarray

Suite

User

Guide

version

edition

Santa

Clara

CA

Affymetrix

BAUGH

L.

H

ILL

A.

B

ROWN

E.

AND

H

UNTER

C.

P.

-LRB-

-RRB-

Quantitative

analysis

of

mRNA

amplification

by

in

vitro

transcription

Nucleic

Acids

Research

B

OLSTAD

B.

I

RIZARRY

R.

STRAND

M.

AND

S

PEED

T.

-LRB-

-RRB-

A

comparison

of

normalization

methods

for

high

density

oligonucleotide

array

data

based

on

variance

and

bias

Bioinformatics

to

appear

D

UDOIT

S.

YANG

Y.

H.

C

ALLOW

M.

J.

AND

S

PEED

T.

P.

-LRB-

-RRB-

Statistical

methods

for

identifying

genes

with

differentialexpression

in

replicated

cDNA

microarray

experiments

Statistica

Sinica

H

ARTEMINK

A.

J.

G

IFFORD

D.

K.

JAAKOLA

T.

S.

AND

YOUNG

R.

A.

-LRB-

-RRB-

Maximum

likelihood

estimation

of

optimal

scaling

factors

for

expression

array

normalization

SPIE

BiOS

H

ILL

A.

H

UNTER

C.

T

SUNG

B.

T

UCKER

K

ELLOGG

G.

expression

in

c.

elegans

Science

AND

B

ROWN

E.

-LRB-

-RRB-

Genomic

analysis

of

gene

H

ILL

A.

A.

B

ROWN

E.

L.

W

HITLEY

M.

Z.

T

UCKER

K

ELLOGG

G.

H

UNTER

C.

P.

AND

S

LONIM

D.

K.

-LRB-

-RRB-

Evaluation

of

normalization

procedures

for

oligonucleotide

array

data

based

on

spiked

cRNA

controls

Genomebiology

H

OLDER

D.

R

AUBERTAS

R.

F.

P

IKOUNIS

V.

B.

S

VETNIK

V.

AND

S

OPER

K.

-LRB-

-RRB-

Statistical

analysis

of

high

density

oligonucleotide

arrays

a

SAFER

approach

Proceedings

of

the

ASA

Annual

Meeting

Atlanta

GA.

H

UBBELL

E.

-LRB-

-RRB-

Estimating

signal

with

next

generation

Affymetrix

software

Gene

Logic

Workshop

on

Low

Level

Analysis

of

AffymetrixGeneChip

R

data

L

I

C.

AND

W

ONG

W.

-LRB-

-RRB-

Model-based

analysis

of

oligonucleotide

arrays

Expression

index

computation

and

outlier

detection

Proceedings

of

the

National

Academy

of

Science

U

S

A

L

OCKHART

D.

J.

D

ONG

H.

B

YRNE

M.

C.

F

OLLETTIE

M.

T.

G

ALLO

M.

V.

C

HEE

M.

S.

M

ITTMANN

M.

WANG

C.

KOBAYASHI

M.

H

ORTON

H.

AND

B

ROWN

E.

L.

-LRB-

-RRB-

Expression

monitoring

by

hybridization

to

high-density

oligonucleotide

arrays

Nature

Biotechnology

NAEF

F.

L

IM

D.

A.

PATIL

N.

AND

M

AGNASCO

M.

O.

-LRB-

-RRB-

From

features

to

expression

High

density

oligonucleotide

array

analysis

revisited

Tech

Report

-LSB-

Received

June

revised

July

accepted

for

publication

July

-RSB-

O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O