BIOINFORMATICS

Structural

bioinformatics

ORIGINAL

PAPER

Vol

no.

pages

doi

Statistical

analysis

of

domains

in

interacting

protein

pairs

Tom

M.

W.

Carlo

Walter

R.

M.

Madan

and

Sarah

A.

Medical

Research

Council

Biostatistics

Unit

Cambridge

UK

Dipartimento

di

Informatica

e

Sistemistica

Universit

di

Pavia

Italy

and

Medical

Research

Council

Laboratory

of

Molecular

Biology

Cambridge

UK

Received

on

June

revised

on

September

accepted

on

October

Advance

Access

publication

October

ABSTRACT

Motivation

Several

methods

have

recently

been

developed

to

analyse

large-scale

sets

of

physical

interactions

between

proteins

in

terms

of

physical

contacts

between

the

constituent

domains

often

with

a

view

to

predicting

new

pairwise

interactions

Our

aim

is

to

combine

genomic

interaction

data

in

which

domain

domain

contacts

are

not

explicitly

reported

with

the

domain-level

structure

of

individual

proteins

in

order

to

learn

about

the

structure

of

interacting

protein

pairs

Our

approach

is

driven

by

the

need

to

assess

the

evidence

for

physical

contacts

between

domains

in

a

statistically

rigorous

way

Results

We

develop

a

statistical

approach

that

assigns

p-values

to

pairs

of

domain

superfamilies

measuring

the

strength

of

evidence

within

a

set

of

protein

interactions

that

domains

from

these

superfamilies

form

contacts

A

set

of

p-values

is

calculated

for

SCOP

superfamily

pairs

based

on

a

pooled

data

set

of

interactions

from

yeast

These

p-values

can

be

used

to

predict

which

domains

come

into

contact

in

an

interacting

protein

pair

This

predictive

scheme

is

tested

against

protein

complexes

in

the

Protein

Quaternary

Structure

-LRB-

PQS

-RRB-

database

and

is

used

to

predict

domain

domain

contacts

within

interacting

protein

pairs

taken

from

our

pooled

data

set

Contact

thomas.nye@mrc-bsu.cam.ac.uk

INTRODUCTION

Proteins

frequently

bind

together

in

pairs

or

larger

complexes

to

take

part

in

biological

processes

Understanding

such

interactions

across

the

entire

genome

is

an

important

goal

with

diverse

implications

about

protein

function

Experimental

techniques

that

test

thousands

of

pairs

of

proteins

within

a

genome

for

their

ability

to

interact

-LRB-

Ito

et

al.

Uetz

et

al.

-RRB-

have

therefore

attracted

considerable

interest

These

large-scale

interaction

assays

produce

no

explicit

information

about

structural

aspects

of

the

interactions

how

the

proteins

come

into

contact

physically

and

the

shapes

they

adopt

Our

aim

in

this

paper

is

to

combine

such

interaction

data

with

information

about

the

structure

of

individual

proteins

at

the

domain

level

in

order

to

learn

about

the

structure

of

protein

complexes

A

number

of

different

experimental

and

computational

techniques

are

available

for

elucidating

the

structure

of

protein

complexes

-LRB-

Russell

et

al.

-RRB-

but

the

complete

three-dimensional

atomic

structure

is

available

only

for

a

limited

range

of

proteins

and

their

To

whom

correspondence

should

be

addressed

complexes

While

such

information

can

be

used

to

predict

the

geometry

of

protein

complexes

of

unknown

structure

-LRB-

Aloy

and

Russell

-RRB-

this

source

of

information

is

severely

limited

by

the

difficulty

of

performing

experiments

to

determine

structures

at

the

atomic

level

Given

the

large

amount

of

genomic

interaction

data

of

the

type

described

above

and

the

fact

that

further

data

are

likely

to

come

available

in

the

future

as

more

experiments

are

performed

techniques

for

extracting

structural

information

from

such

data

are

important

In

addition

while

the

complete

three-dimensional

structure

is

not

available

for

the

majority

of

proteins

it

is

often

possible

to

elucidate

the

structure

of

a

protein

at

the

domain

level

by

sequence

homology

alone

A

number

of

approaches

to

analysing

and

predicting

protein

interactions

in

terms

of

constituent

domains

have

recently

been

published

The

basic

idea

is

to

work

within

some

classification

of

protein

domains

and

identify

pairs

of

domain

superfamilies

-LRB-

or

other

such

groupings

-RRB-

that

occur

frequently

in

interacting

protein

pairs

within

some

large-scale

interaction

data

set

This

information

can

then

be

used

to

make

predictions

about

new

groups

of

proteins

that

might

interact

and

also

indicate

the

domains

which

might

be

contacting

each

other

in

these

proteins

A

common

approach

-LRB-

Kim

et

al.

Ng

et

al.

Sprinzak

and

Margalit

-RRB-

has

been

to

score

domain

pairs

according

to

how

frequently

the

pair

occurs

in

interacting

protein

pairs

one

in

each

protein

relative

to

the

abundance

of

the

domains

across

the

proteome

However

there

is

little

consensus

over

how

such

scores

should

be

calculated

and

a

number

of

statistical

issues

arise

is

it

meaningful

to

compare

the

scores

for

different

domain

pairs

directly

How

would

the

scores

be

distributed

if

the

domain

composition

of

proteins

had

no

influence

on

interaction

A

more

rigorous

alternative

approach

due

to

Deng

et

al.

-LRB-

-RRB-

employs

a

maximum

likelihood

method

to

estimate

a

probability

of

interaction

for

every

domain

pair

This

involves

using

the

EM

algorithm

to

maximize

the

likelihood

in

a

space

with

potentially

thousands

of

dimensions

This

paper

presents

an

approach

to

pairwise

protein

interaction

aimed

at

addressing

the

statistical

issues

mentioned

above

However

unlike

existing

approaches

-LRB-

Deng

et

al.

Sprinzak

and

Margalit

-RRB-

our

aim

is

to

predict

the

most

likely

pair

of

domains

mediating

a

given

protein

interaction

rather

than

predicting

new

protein

interactions

Given

a

pair

of

proteins

that

are

known

to

interact

the

ability

to

predict

structural

aspects

of

this

interaction

would

be

of

great

biological

value

Aloy

et

al.

-LRB-

-RRB-

considered

a

similar

structural

prediction

problem

but

at

a

larger

scale

predicting

Published

by

Oxford

University

Press

T.M.W.Nye

et

al.

protein

protein

contacts

within

complexes

containing

more

than

two

proteins

The

approach

we

adopt

has

the

following

form

A

simple

model

of

contacts

between

protein

domains

is

used

to

assign

a

score

to

superfamily

pairs

Throughout

the

paper

the

term

contact

is

used

to

signify

physical

binding

between

two

domains

in

separate

proteins

Rather

than

comparing

scores

for

different

superfamily

pairs

directly

a

data-simulation

method

is

used

to

generate

a

p-value

for

the

observed

score

of

each

superfamily

pair

measuring

the

strength

of

evidence

for

the

role

played

by

the

superfamily

pair

in

protein

interaction

We

generate

a

set

of

p-values

for

SCOP

superfamily

pairs

based

on

a

set

of

protein

interactions

obtained

by

pooling

together

several

data

sets

of

interactions

from

yeast

The

ranked

list

of

superfamily

pairs

we

obtain

can

be

used

to

predict

which

domains

come

into

contact

in

an

interacting

protein

pair

it

is

natural

to

expect

that

the

pair

with

the

lowest

p-value

should

form

a

contact

We

test

this

idea

by

extracting

contacts

between

proteins

in

complexes

in

the

Protein

Quaternary

Structure

-LRB-

PQS

-RRB-

database

-LRB-

Henrick

and

Thornton

-RRB-

and

comparing

these

contacts

with

our

predictions

We

also

predict

where

contacts

occur

between

the

interacting

protein

pairs

in

our

pooled

set

of

yeast

interactions

and

show

how

our

analysis

expands

the

repertoire

of

superfamily

interactions

found

in

the

PQS

are

not

used

since

they

do

not

contain

information

about

which

proteins

are

in

physical

contact

and

the

evidence

they

contain

for

domain

domain

contacts

is

therefore

much

weaker

More

details

about

the

data

used

are

given

in

Section

More

about

domains

Experimental

data

Broadly

speaking

experiments

that

identify

protein

interactions

fall

into

two

categories

low

throughput

approaches

that

test

for

interactions

between

a

limited

selection

of

proteins

and

high

throughput

approaches

that

survey

a

large

part

of

the

proteome

All

such

techniques

are

affected

by

experimental

errors

with

high

throughput

approaches

generally

more

affected

Experimental

techniques

can

also

be

classified

as

to

whether

they

identify

interacting

protein

pairs

or

larger

complexes

of

proteins

but

in

this

paper

we

deal

only

with

pairwise

interactions

The

yeast

two-hybrid

technique

is

a

high

throughput

approach

that

has

been

used

to

screen

the

yeast

genome

for

pairwise

interactions

by

two

different

groups

-LRB-

Ito

et

al.

Uetz

et

al.

-RRB-

The

advantage

of

screening

the

entire

genome

-LRB-

or

at

least

a

large

part

of

it

-RRB-

for

interactions

is

offset

by

a

high

rate

of

experimental

error

and

it

is

vital

that

the

false

positive

rate

is

included

in

any

analysis

of

the

data

The

potential

benefits

and

drawbacks

of

such

high

throughput

techniques

have

generated

intense

interest

see

Legrain

et

al.

-LRB-

-RRB-

and

von

Mering

et

al.

-LRB-

-RRB-

for

reviews

of

this

area

and

Sprinzak

et

al.

-LRB-

-RRB-

for

estimates

of

the

false

positive

rate

Given

a

list

of

interactions

obtained

from

such

experiments

absence

of

a

protein

pair

from

the

list

does

not

necessarily

mean

that

the

two

proteins

do

not

interact

it

could

be

that

interaction

of

the

pair

in

question

was

not

tested

in

the

experimental

set-up

This

false

negative

probability

has

to

be

accounted

for

when

making

statistical

inference

based

on

absence

from

the

list

of

interactions

The

experimental

data

used

in

this

paper

all

concern

the

proteome

of

Saccaromyces

cerevisiae

Three

data

sets

of

interactions

are

used

interactions

observed

in

Ito

et

al.

-LRB-

-RRB-

and

Uetz

et

al.

-LRB-

-RRB-

yeast

two-hybrid

experiments

together

with

a

set

of

interactions

from

the

MIPS

database

-LRB-

Mewes

et

al.

-RRB-

The

MIPS

database

contains

interactions

determined

by

a

variety

of

experimental

methods

many

of

which

are

focussed

on

individual

interactions

rather

than

largescale

interaction

determination

Datasets

of

larger

protein

complexes

Domains

are

structural

subunits

of

proteins

that

can

be

thought

of

as

building

blocks

that

are

conserved

during

evolution

Proteins

can

consist

of

a

single

domain

or

more

frequently

as

a

combination

of

several

domains

in

prokaryotes

about

one-third

of

all

proteins

are

single

domain

while

in

eukaryotes

the

fraction

is

about

-LRB-

Apic

et

al.

-RRB-

Nature

is

able

to

construct

a

vast

array

of

different

proteins

by

combining

domains

and

the

present-day

variety

of

domains

is

believed

to

have

evolved

from

a

relatively

small

number

of

ancestral

gene

sequences

Domains

which

are

related

to

each

other

by

descent

from

a

common

ancestor

are

said

to

be

homologous

and

can

be

grouped

together

to

form

a

superfamily

Domains

in

the

same

superfamily

usually

preserve

the

same

three-dimensional

conformation

if

not

function

but

their

sequences

may

have

diverged

considerably

during

evolution

a

phenomenon

called

remote

homology

We

regard

the

SCOP

database

-LRB-

Murzin

et

al.

-RRB-

as

the

standard

for

classifying

domains

in

this

way

and

we

work

throughout

with

the

SCOP

classification

It

is

often

possible

to

determine

a

set

of

constituent

domain

superfamilies

for

a

protein

given

its

amino

acid

sequence

The

SUPERFAMILY

library

-LRB-

Gough

et

al.

-RRB-

is

a

facility

coupled

to

the

SCOP

classification

used

in

this

study

to

perform

such

assignments

It

consists

of

a

library

of

hidden

Markov

models

-LRB-

HMMs

-RRB-

algorithms

that

have

been

trained

to

detect

remote

homology

to

the

SCOP

superfamilies

in

protein

sequences

Given

an

amino

acid

sequence

it

produces

a

string

of

superfamily

labels

that

represents

the

sequence

of

domains

in

the

protein

This

string

may

contain

gaps

for

regions

where

no

superfamily

assignment

was

made

and

the

process

may

fail

to

assign

any

superfamilies

to

certain

proteins

Proteins

that

were

not

assigned

any

superfamilies

were

removed

from

our

analysis

and

gaps

in

the

domain

assignments

were

ignored

This

can

introduce

a

bias

if

such

regions

are

involved

in

interaction

as

described

further

in

Section

More

information

on

obtaining

domain

assignments

is

available

from

While

domains

within

the

same

superfamily

are

related

by

a

common

ancestor

it

is

possible

that

during

the

course

of

evolution

their

functions

may

have

diverged

Similarly

the

repertoire

of

interaction

partners

for

each

domain

will

not

necessarily

be

conserved

If

we

observe

domains

from

a

particular

superfamily

pair

coming

into

contact

in

a

given

protein

interaction

it

does

not

follow

that

in

other

protein

pairs

domains

from

this

superfamily

pair

will

always

form

contacts

However

our

statistical

approach

is

able

to

take

this

into

account

the

p-values

we

generate

measure

the

evidence

for

the

role

played

by

each

superfamily

pair

in

protein

interaction

When

an

interaction

has

been

poorly

conserved

between

the

members

of

two

superfamilies

this

superfamily

pair

will

be

less

significant

and

assigned

a

higher

p-value

METHODS

This

section

describes

the

algorithm

used

to

assign

p-values

to

superfamily

pairs

and

the

procedure

for

predicting

domain

domain

contacts

between

proteins

Section

introduces

our

notation

and

assumptions

The

algorithm

used

to

compute

the

p-values

is

described

in

sections

and

and

is

illustrated

schematically

in

Figure

In

Section

we

provide

details

of

the

data

Statistical

analysis

of

interacting

protein

pairs

Fig.

A

schematic

representation

of

the

algorithm

to

generate

p-values

Shaded

symbols

represent

superfamilies

set

used

to

generate

the

p-values

Finally

Section

describes

the

scheme

for

predicting

domain

domain

contacts

Notation

and

assumptions

Consider

the

complete

set

of

proteins

expressed

in

some

organism

and

let

P

denote

the

set

of

all

-LRB-

unordered

-RRB-

protein

pairs

Let

D

denote

the

set

of

all

possible

pairs

of

domain

superfamilies

Note

that

P

includes

proteins

paired

with

themselves

representing

the

possibility

of

interaction

between

identical

proteins

Similarly

D

includes

domain

superfamilies

paired

with

themselves

Throughout

we

often

use

the

symbol

i

to

denote

an

element

of

P

and

j

to

denote

an

element

of

D.

Each

protein

p

is

assumed

to

have

a

domain

assignment

a

string

-LRB-

-RRB-

where

each

Ak

is

a

domain

superfamily

Our

analysis

does

not

acknowledge

any

uncertainty

in

this

assignment

a

valid

assumption

on

account

of

the

high

accuracy

of

the

domain

HMMs

We

will

write

A

p

to

denote

that

protein

p

contains

a

domain

of

superfamily

A.

Given

a

protein

pair

i

-LRB-

p

q

-RRB-

and

superfamily

pair

j

-LRB-

A

B

-RRB-

let

Nij

be

the

number

of

distinct

ways

of

having

p

bound

to

q

via

a

contact

between

a

domain

in

superfamily

A

and

a

domain

in

superfamily

B.

Of

course

Nij

if

and

only

if

A

p

and

B

q

or

B

p

and

A

q

We

make

the

simplifying

assumption

that

each

pair

of

proteins

i

P

either

interacts

or

not

thereby

ignoring

issues

such

as

the

difference

between

T.M.W.Nye

et

al.

stable

and

transient

interactions

or

interactions

that

require

specific

cellular

conditions

Let

I

P

denote

the

set

of

such

interacting

pairs

In

its

simplest

form

the

experimental

data

is

a

list

E

P

of

protein

pairs

that

have

been

observed

to

interact

It

is

important

that

experimental

error

and

incompleteness

of

the

data

are

taken

into

account

for

which

we

assume

fixed

probabilities

of

false

positive

and

false

negative

f

Pr

-LRB-

i

I

i

E

-RRB-

and

f

Pr

-LRB-

i

I

i

E

-RRB-

In

addition

to

the

expected

number

of

contacts

Ej

we

also

need

to

consider

the

total

number

of

different

possible

contacts

of

type

j

within

the

proteome

defined

by

Nj

Nij

iP

Next

we

calculate

a

measure

of

interaction

for

each

superfamily

pair

j

in

-LRB-

j

-RRB-

the

following

way

A

array

xrs

is

defined

by

Ej

Nj

Ej

-LRB-

j

-RRB-

-LRB-

j

-RRB-

Here

Pr

-LRB-

U

V

-RRB-

denotes

the

conditional

probability

of

an

event

U

given

another

event

V

It

follows

that

the

probability

that

protein

pair

i

interacts

given

the

experimental

data

E

is

given

by

Pr

-LRB-

i

I

E

-RRB-

f

f

if

i

E

if

i

E.

-LRB-

-RRB-

k

j

-LRB-

j

-RRB-

Ek

-LRB-

Nk

Ek

-RRB-

k

j

-LRB-

j

-RRB-

and

the

log

odds

ratio

s

-LRB-

j

-RRB-

for

the

array

is

calculated

s

-LRB-

j

-RRB-

log

log

log

log

A

large

value

for

s

-LRB-

j

-RRB-

is

obtained

when

the

superfamily

pair

j

has

a

relatively

large

expected

number

of

contacts

in

comparison

with

the

other

superfamily

pairs

Measures

of

interaction

other

than

the

log

odds

ratio

can

be

used

but

in

practice

the

choice

of

statistic

was

observed

to

have

a

small

affect

on

the

p-values

Recall

that

for

each

superfamily

pair

j

we

are

interested

in

testing

the

hypothesis

Hj

that

presence

of

the

superfamily

pair

in

a

protein

pair

has

no

effect

on

whether

the

two

proteins

interact

If

protein

interaction

is

not

affected

by

the

domain

content

of

the

proteins

-LRB-

i.e.

if

the

global

hypothesis

H

were

true

-RRB-

then

we

can

create

a

new

set

of

proteins

in

which

the

domains

have

been

shuffled

between

and

within

proteins

that

interact

in

exactly

the

same

way

as

the

original

set

The

precise

nature

of

this

shuffling

is

discussed

below

It

is

important

to

note

that

the

while

the

domains

are

shuffled

the

network

of

interactions

between

proteins

remains

fixed

By

shuffling

the

domains

around

in

this

way

to

create

n

replicate

data

sets

and

re-calculating

the

statistics

-LRB-

j

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

s

-LRB-

j

-RRB-

for

each

set

we

generate

a

set

of

statistics

sn

for

each

superfamily

pair

j

D

that

represents

a

sample

from

the

distribution

of

the

statistic

s

-LRB-

j

-RRB-

under

H

By

counting

the

number

of

times

the

simulated

-LRB-

j

-RRB-

statistics

exceed

the

observed

statistics

we

can

obtain

a

p-value

for

the

observed

data

pj

-LRB-

number

of

times

that

the

statistics

n

-LRB-

j

sn

-RRB-

exceed

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

-LRB-

j

-RRB-

More

generally

we

may

have

several

different

sets

of

experimentally

observed

interacting

pairs

En

each

with

a

different

false

posit

ive

rate

fn

A

simple

approach

to

combining

these

is

to

take

the

probability

of

false

positive

for

a

particular

pair

i

Ek

to

be

the

lowest

error

rate

across

data

sets

containing

that

pair

The

observed

sets

of

interactions

that

we

consider

overlap

very

little

so

this

is

a

valid

assumption

Equation

-LRB-

-RRB-

is

then

replaced

by

Pr

-LRB-

i

I

E

-RRB-

min

-LCB-

fk

i

Ek

-RCB-

if

i

f

if

i

Ek

Ek

-LRB-

-RRB-

Estimates

for

the

probabilities

of

false

positive

and

negative

in

our

data

are

given

in

Section

Assigning

p-values

to

superfamily

pairs

For

each

superfamily

pair

j

D

we

want

to

test

the

hypothesis

Hj

that

presence

of

the

superfamily

pair

in

a

protein

pair

does

not

affect

whether

the

two

proteins

interact

We

also

consider

the

global

hypothesis

H

Hj

that

interaction

is

entirely

unrelated

to

the

domain

architectures

of

proteins

In

order

to

test

these

hypotheses

a

statistic

is

attached

to

each

superfamily

pair

j

that

reflects

the

strength

of

evidence

present

in

E

that

j

affects

protein

interaction

A

p-value

is

calculated

for

each

statistic

by

simulating

data

drawn

from

the

global

hypothesis

H

We

now

explain

this

procedure

in

detail

The

first

step

is

to

consider

the

expected

number

of

contacts

between

domains

of

type

j

under

the

H

given

the

observed

set

of

interactions

E

In

order

to

compute

this

we

make

the

following

assumption

when

a

pair

of

proteins

i

-LRB-

p

q

-RRB-

interact

they

do

so

by

bringing

a

single

domain

from

p

into

contact

with

a

single

domain

from

q.

Biological

evidence

suggests

that

contacts

may

in

fact

occur

between

several

domains

in

an

interacting

protein

pair

but

this

simplifying

assumption

can

nonetheless

be

used

to

extract

useful

information

Given

the

experimental

data

E

under

the

H

and

the

assumption

above

the

expected

number

of

contacts

Ej

between

domains

of

type

j

across

the

entire

proteome

is

given

by

Ej

iP

The

p-value

pj

represents

the

probability

of

observing

the

data

if

the

hypothesis

Hj

is

true

Shuffling

algorithms

Pr

-LRB-

i

interacts

via

contact

of

type

j

E

-RRB-

Pr

-LRB-

i

interacts

E

-RRB-

iP

Pr

-LRB-

contact

is

of

type

j

i

interacts

-RRB-

The

total

number

of

possible

contacts

within

pair

i

is

global

Pr

-LRB-

contact

is

of

type

j

i

interacts

-RRB-

It

follows

that

Ej

iP

k

Nik

so

under

the

Nij

k

Nik

-LRB-

-RRB-

Pr

-LRB-

i

I

E

-RRB-

Nij

k

Nik

The

first

factor

in

this

expression

depends

on

E

and

is

given

by

Equation

-LRB-

-RRB-

The

second

factor

depends

only

on

the

domain

architectures

of

the

proteins

This

section

describes

in

some

detail

the

algorithm

used

to

shuffle

domain

superfamilies

The

exact

way

in

which

the

domain

superfamilies

are

shuffled

at

each

stage

reflects

different

implicit

assumptions

in

the

hypothesis

H

The

simplest

method

is

random

permutation

of

the

domain

superfamilies

as

follows

A

string

is

formed

by

concatenating

the

domain

assignments

of

all

the

proteins

A

random

permutation

is

applied

to

the

string

and

the

shuffled

string

is

used

to

re-assign

domain

superfamilies

to

proteins

This

maintains

the

marginal

frequencies

of

each

superfamily

within

the

data

set

as

well

as

the

number

of

domains

found

in

each

protein

However

in

nature

certain

consecutive

superfamily

pairs

are

over-represented

in

the

domain

architectures

of

proteins

-LRB-

Apic

et

al.

-RRB-

indeed

this

tends

to

be

the

rule

rather

than

exception

For

example

the

motif

A

B

may

occur

more

frequently

in

the

domain

architectures

from

an

entire

proteome

than

would

be

expected

given

the

marginal

frequency

of

the

two

superfamilies

A

and

B

It

is

desirable

to

include

this

feature

when

testing

the

significance

of

superfamily

pairs

This

is

achieved

by

shuffling

the

superfamilies

at

each

stage

using

an

algorithm

that

partially

maintains

the

marginal

frequency

of

consecutive

superfamilies

across

the

proteome

The

first

step

is

to

shuffle

the

domains

in

the

single-domain

proteins

using

the

permutation

method

described

above

Next

a

string

is

formed

by

concatenating

the

domain

architectures

of

all

Statistical

analysis

of

interacting

protein

pairs

Table

Interaction

data

sets

f

-LRB-

-RRB-

was

found

in

The

false

negative

rate

f

was

estimated

as

follows

Pr

-LRB-

i

I

i

E

-RRB-

Pr

-LRB-

i

I

E

-RRB-

Pr

-LRB-

i

E

-RRB-

I

I

E

P

E

Dataset

Ito

core

non-core

Uetz

MIPS

N

NR

806 3669 948 1281

240 859 344 630

N

is

the

number

of

interactions

in

the

data

set

while

NR

is

the

number

of

interactions

that

remain

following

removal

of

ORFs

that

can

not

be

assigned

any

domains

f

is

the

false

positive

rate

The

core

Ito

interactions

are

supported

by

a

larger

number

of

observations

than

the

non-core

Ito

-LRB-

core

-RRB-

-LRB-

core

-RRB-

-LRB-

core

-RRB-

MIPS

where

E

is

the

complement

of

E

In

fact

E

is

small

compared

to

P

and

so

can

be

ignored

in

the

denominator

There

are

ORFs

in

the

yeast

genome

and

approximately

interactions

giving

I

P

The

estimate

of

the

total

number

of

interactions

I

is

necessarily

approximate

and

is

taken

from

Deng

et

al.

-LRB-

-RRB-

and

Legrain

et

al.

-LRB-

-RRB-

The

correction

term

I

E

can

be

estimated

using

the

probabilities

of

false

positive

to

give

a

final

false

negative

estimate

of

f

for

the

combined

data

set

In

order

to

reduce

the

computational

burden

p-values

were

calculated

only

for

those

superfamily

pairs

for

which

there

was

some

evidence

of

interaction

in

the

pooled

data

set

each

possible

superfamily

pair

was

included

provided

it

was

present

in

at

least

one

experimentally

observed

interaction

Let

S

denote

this

set

of

superfamily

pairs

Using

the

pooled

set

of

interactions

S

contained

superfamily

pairs

Over

iterations

of

the

algorithm

were

performed

to

compute

the

p-values

Predicting

contacts

-LRB-

core

-RRB-

Uetz

Fig.

Numbers

of

interactions

in

the

data

sets

following

removal

of

ORFs

that

were

not

assigned

any

domains

We

expect

that

in

any

protein

pair

that

is

known

to

interact

the

domains

belonging

to

the

superfamily

pair

with

the

lowest

p-value

are

most

likely

to

form

a

contact

This

type

of

prediction

was

tested

out

on

protein

complexes

in

the

PQS

database

-LRB-

Henrick

and

Thornton

-RRB-

for

each

pair

of

interacting

proteins

we

predict

a

domain

domain

contact

and

compare

this

against

the

true

three-dimensional

configuration

The

PQS

is

an

Internet

resource

that

makes

available

coordinates

for

likely

quaternary

states

for

structures

contained

in

the

Brookhaven

Protein

Data

Bank

-LRB-

PDB

-RRB-

that

were

determined

by

X-ray

crystallography

Contacts

between

proteins

in

the

PQS

were

extracted

by

analysing

the

positions

of

constituent

atoms

Predictions

were

made

for

interacting

protein

pairs

that

satisfied

the

following

constraints

At

least

one

of

the

proteins

must

contain

more

than

one

domain

If

this

is

not

the

case

predicting

which

domains

come

into

contact

is

trivial

Both

proteins

must

only

contain

domains

from

superfamilies

that

are

represented

in

the

yeast

genome

Presence

of

domains

from

superfamilies

not

found

in

yeast

would

bias

the

results

In

addition

we

require

that

at

least

one

of

the

possible

contacts

between

the

proteins

is

represented

in

the

set

S

defined

in

Section

If

all

the

possible

contacts

lie

outside

S

then

there

is

no

evidence

in

E

to

support

any

of

the

contacts

so

attempting

to

make

predictions

is

futile

There

are

interacting

protein

pairs

in

the

PQS

satisfying

conditions

and

and

of

these

also

satisfy

condition

Predictions

were

also

made

for

this

data

set

using

the

scores

of

Sprinzak

and

Margalit

-LRB-

-RRB-

and

Deng

et

al.

-LRB-

-RRB-

The

scores

were

obtained

by

training

on

the

pooled

data

set

described

in

Section

The

Sprinzak-score

for

each

superfamily

pair

can

be

calculated

very

easily

whereas

the

Deng-score

is

computed

via

an

involved

likelihood

maximization

procedure

When

making

domain

domain

contact

predictions

based

on

each

of

these

scores

the

domain

pair

with

the

highest

score

was

taken

as

the

predicted

contact

Note

that

when

a

protein

contains

a

number

of

domains

from

the

same

superfamily

several

potential

contacts

may

be

assigned

the

same

p-value

-LRB-

or

score

-RRB-

and

our

predictive

scheme

is

unable

to

distinguish

between

these

When

the

minimum

p-value

occurs

for

several

potential

contacts

in

this

way

we

simply

choose

one

of

them

at

random

Domain

contact

predictions

were

also

made

using

the

p-values

for

interacting

protein

pairs

satisfying

condition

above

taken

from

our

pooled

set

of

training

data

In

total

of

the

interactions

satisfied

this

condition

For

many

of

the

yeast

ORFs

there

is

no

known

protein

structure

and

even

less

is

known

about

the

structure

of

binary

complexes

These

predictions

therefore

provide

novel

information

as

to

how

the

interactions

could

be

mediated

the

multi-domain

proteins

The

letters

of

this

string

are

shuffled

in

such

a

way

as

to

preserve

their

marginal

frequencies

together

with

the

marginal

frequency

of

consecutive

letter

pairs

via

an

algorithm

described

in

Fitch

-LRB-

-RRB-

The

shuffled

string

is

then

used

to

re-assign

domains

to

the

multi-domain

proteins

Data

and

simulation

The

domain

assignments

for

the

yeast

open

reading

frames

-LRB-

ORFs

-RRB-

were

obtained

using

the

SUPERFAMILY

library

release

number

After

eliminating

ORFs

that

were

not

assigned

any

superfamilies

ORFs

remained

containing

a

total

of

different

domain

superfamilies

Of

these

ORFs

were

assigned

a

single

domain

superfamily

while

were

assigned

more

than

one

superfamily

Table

summarizes

the

interaction

data

sets

The

Ito

data

set

comprises

two

parts

-LRB-

core

and

non-core

-RRB-

with

stronger

levels

of

evidence

for

interactions

in

the

core

part

The

MIPS

data

was

taken

from

the

pairwise

interactions

released

on

August

and

consists

of

all

physical

interactions

obtained

via

methods

other

than

yeast

two-hybrid

The

second

column

of

Table

lists

the

number

of

interactions

that

remain

after

eliminating

ORFs

that

were

not

assigned

any

superfamilies

The

approximate

estimates

of

the

false

positive

rates

are

taken

from

Sprinzak

et

al.

-LRB-

-RRB-

Figure

shows

the

extent

to

which

the

data

sets

overlap

-LRB-

after

eliminating

ORFs

-RRB-

The

simulation

to

generate

p-values

was

performed

using

a

set

of

interactions

obtained

by

pooling

together

the

Uetz

data

set

the

MIPS

data

set

and

the

Ito

data

set

-LRB-

combining

core

and

non-core

-RRB-

The

probability

of

false

positive

for

each

interaction

was

taken

to

be

the

minimum

for

each

of

the

data

sets

it

T.M.W.Nye

et

al.

Fig.

An

extract

of

the

network

of

SCOP

superfamily

interactions

Interactions

between

domains

on

separate

chains

within

the

PQS

are

shown

as

blue

edges

Red

edges

represent

superfamily

pairs

with

p-value

below

a

certain

threshold

-LRB-

-RRB-

Edges

satisfying

both

criteria

are

marked

in

green

Loops

correspond

to

interactions

between

domains

in

the

same

superfamily

Superfamilies

of

enzymatic

domains

are

shaded

yellow

RESULTS

Results

can

be

downloaded

from

A

list

of

superfamily

pairs

and

their

p-values

is

available

from

this

website

together

with

the

domain

domain

contact

predictions

for

our

pooled

set

of

interaction

data

Analysis

of

interacting

protein

pairs

in

the

PQS

reveals

contacts

between

domains

from

different

superfamily

pairs

-LRB-

restricting

to

superfamilies

represented

in

the

yeast

genome

-RRB-

Our

analysis

of

the

experimental

genomic

interaction

data

suggests

contacts

between

a

set

of

superfamily

pairs

in

the

following

way

We

make

the

basic

assumption

that

superfamily

pairs

assigned

a

sufficiently

low

p-value

form

contacts

and

extract

a

list

of

pairs

by

imposing

a

p-value

threshold

The

p-value

threshold

was

chosen

to

ensure

a

false

discovery

rate

-LRB-

FDR

-RRB-

of

we

expect

of

superfamily

pairs

falling

below

the

threshold

not

to

be

significant

-LRB-

see

Benjamini

and

Hochberg

for

more

information

on

the

FDR

-RRB-

The

FDR

of

corresponds

to

a

p-value

threshold

of

and

superfamily

pairs

lie

below

this

threshold

Seventy-three

of

these

pairs

occur

as

contacts

in

the

PQS

Figure

shows

a

small

part

of

the

network

of

superfamily

interactions

indicating

how

our

analysis

extends

the

repertoire

of

contacts

represented

in

the

PQS

Figure

shows

the

results

of

testing

the

predictive

scheme

described

in

Section

on

the

PQS

Each

interacting

protein

pair

in

the

PQS

satisfying

our

constraints

is

classified

according

to

the

number

of

potential

contacts

-LRB-

For

example

a

protein

containing

two

domains

interacting

with

a

three-domain

protein

has

six

potential

contacts

-RRB-

Since

contacts

often

occur

between

several

different

domain

pairs

within

each

interacting

protein

pair

in

the

PQS

if

we

simply

pick

one

potential

contact

at

random

there

is

a

certain

probability

that

this

will

be

observed

as

a

true

contact

The

expected

success

rate

by

picking

a

potential

contact

at

random

is

shown

in

the

figure

together

with

the

success

rates

obtained

using

the

Sprinzak-score

the

Deng-score

and

the

p-values

From

the

figure

it

can

be

seen

that

the

expert

prediction

methods

do

not

always

outperform

naive

prediction

at

random

-LRB-

e.g.

in

the

case

of

four

potential

contacts

-RRB-

Moreover

in

some

cases

there

is

insufficient

test

data

for

meaningful

comparisons

between

the

methods

to

Statistical

analysis

of

interacting

protein

pairs

Fig.

Domain

domain

contact

prediction

results

The

results

are

broken

down

according

to

the

potential

number

of

domain

domain

contacts

available

between

protein

pairs

in

the

PQS

database

and

the

number

of

protein

pairs

within

each

such

category

is

shown

at

the

bottom

of

the

figure

The

proportion

of

protein

pairs

for

which

four

different

prediction

methods

correctly

predict

a

domain

domain

contact

is

shown

in

the

main

graph

It

is

often

observed

in

the

PQS

that

several

different

domain

pairs

are

in

contact

within

each

interacting

protein

pair

Any

potential

contact

picked

at

random

therefore

has

some

probability

of

being

confirmed

as

a

contact

in

the

PQS

and

this

baseline

success

rate

is

shown

by

the

hatched

bars

The

other

bars

correspond

to

prediction

using

the

Sprinzak-score

Deng-score

and

lowest

p-value

as

described

in

Section

The

error

bars

correspond

to

a

confidence

interval

based

on

a

binomial

distribution

assumption

be

made

For

interacting

proteins

with

two

or

six

potential

contacts

the

p-value

prediction

method

does

not

perform

as

well

as

prediction

using

the

Deng

or

Sprinzak

scores

although

in

the

case

of

six

potential

contacts

the

evidence

for

this

is

limited

However

as

the

number

of

potential

contacts

increases

and

as

the

prediction

problem

becomes

harder

the

p-value

method

outperforms

the

other

methods

In

particular

in

the

case

of

nine

potential

contacts

there

are

test

pairs

in

the

PQS

and

the

p-value

method

performs

significantly

better

than

prediction

based

on

the

Sprinzak

or

Deng

scores

It

should

be

noted

that

the

ability

to

make

predictions

of

this

kind

is

limited

by

three

factors

It

is

a

well-known

feature

of

the

genomic

protein

protein

interaction

data

sets

that

they

explore

a

relatively

small

region

of

the

vast

space

of

possible

interactions

-LRB-

Legrain

et

al.

-RRB-

and

so

making

predictions

on

the

basis

of

this

data

will

be

limited

though

in

the

future

this

coverage

will

probably

improve

For

example

in

of

the

PQS

protein

pairs

with

four

potential

contacts

of

the

true

contacts

arose

as

a

possible

contact

in

the

genomic

data

sets

Secondly

despite

the

constraints

we

impose

on

PQS

entries

the

protein

complexes

for

which

we

make

predictions

are

not

representative

of

all

such

complexes

in

yeast

owing

to

the

nature

and

constraints

of

crystallographic

experiments

Thirdly

when

the

proteins

involve

a

number

of

repeated

domains

several

different

potential

contacts

will

receive

the

same

p-value

and

our

predictive

scheme

is

unable

to

distinguish

between

these

This

also

applies

to

the

Sprinzak

and

Deng

scores

In

addition

to

testing

the

predictive

approach

against

the

PQS

domain

domain

contact

predictions

were

made

for

protein

interactions

in

our

pooled

data

set

for

which

more

than

one

domain

domain

contact

was

possible

These

predictions

provide

basic

structural

information

for

a

large

number

of

protein

interactions

for

which

such

information

has

previously

not

been

available

Selected

examples

are

shown

in

Figure

Note

that

these

predictions

condition

on

each

protein

pair

interacting

if

the

two

proteins

truly

interact

our

prediction

indicates

the

domain

domain

contact

most

favoured

by

our

analysis

but

the

pooled

data

set

also

contains

false

positives

However

it

is

natural

to

assume

that

protein

pairs

are

more

likely

to

interact

when

the

lowest

p-value

is

small

A

total

of

of

the

interactions

have

the

smallest

p-value

below

the

threshold

of

fixed

above

and

this

information

is

included

in

the

results

on

our

website

T.M.W.Nye

et

al.

Fig.

Schematic

diagram

showing

predicted

domain

domain

contacts

for

pairwise

interactions

selected

from

the

Uetz

and

MIPS

data

sets

The

different

shapes

correspond

to

SCOP

superfamilies

while

regions

of

proteins

that

were

not

assigned

SCOP

superfamilies

are

shown

as

black

lines

Proteins

are

displayed

with

the

N-terminal

on

the

left

The

arrows

indicate

the

predicted

domain

domain

contact

for

each

interaction

-LRB-

A

-RRB-

Predicted

contacts

between

a

Cullin

repeat

domain

on

the

protein

and

domains

from

the

dimerization

domainlike

superfamily

on

the

proteins

and

These

protein

interactions

are

from

the

Uetz

data

set

and

are

supported

by

other

experiments

-LSB-

Yamanaka

A.

et

al.

-LRB-

-RRB-

Curr

Biol

-LRB-

-RRB-

and

Fritz

S.

et

al.

-LRB-

-RRB-

Mol

Biol

Cell

-LRB-

-RRB-

-RSB-

Domains

from

these

superfamilies

are

known

to

interact

in

a

human

protein

complex

-LSB-

Zheng

N.

et

al.

-LRB-

-RRB-

Nature

-LRB-

-RRB-

-RSB-

and

a

structure

for

this

interaction

is

known

-LRB-

PDB

identifier

-RRB-

-LRB-

B

-RRB-

Predicted

contacts

between

an

ATPase

domain

and

domain

I

of

DNA

repair

protein

MutS

Four

pairwise

interactions

are

shown

and

These

proteins

are

involved

in

mismatch

repair

in

meiosis

and

mitosis

specifically

single-base

and

insertion

deletion

mismatch

repair

The

mode

of

interaction

is

unknown

and

our

predictions

are

included

as

a

suggestion

for

experimental

verification

DISCUSSION

This

paper

proposes

a

methodology

for

the

analysis

of

large

sets

of

protein

interaction

data

from

genomic

experiments

in

terms

of

the

constituent

domains

within

the

proteins

The

main

motivation

behind

the

methodology

is

the

need

to

assess

the

evidence

in

such

data

sets

for

physical

contact

between

domains

in

a

statistically

rigorous

way

In

particular

unlike

existing

approaches

in

the

literature

our

method

allows

domain

pairs

to

be

ranked

in

terms

of

evidence

of

contact

by

using

a

rigorous

statistical

measure

of

evidence

the

p-value

A

sophisticated

simulation

technique

is

necessary

to

generate

these

Statistical

analysis

of

interacting

protein

pairs

p-values

since

the

usual

statistical

association

tests

do

not

have

an

explicit

asymptotic

distribution

due

to

the

complexity

of

the

data

Our

methodology

allows

for

observational

uncertainty

specifically

false

positive

and

false

negative

rates

in

interaction

experiments

Estimates

for

these

rates

are

incorporated

in

the

analysis

with

the

possibility

of

merging

data

reflecting

different

degrees

of

error

in

the

same

analysis

By

imposing

a

p-value

threshold

we

extracted

a

set

of

superfamily

pairs

that

play

a

statistically

significant

role

in

protein

interaction

The

p-value

threshold

was

chosen

in

such

a

way

as

to

control

the

false

discovery

rate

-LRB-

Benjamini

and

Hochberg

-RRB-

i.e.

the

expected

number

of

pairs

incorrectly

included

on

the

list

Under

the

assumption

that

domains

from

these

superfamily

pairs

form

physical

contacts

we

have

demonstrated

how

large-scale

interaction

data

sets

extend

the

collection

of

superfamily

contacts

observed

in

the

PQS

database

We

have

also

tested

a

simple

method

for

predicting

domain

domain

contacts

between

interacting

proteins

on

the

basis

of

the

p-values

Predictions

were

made

for

interacting

protein

pairs

in

the

PQS

for

which

the

contacts

are

known

Prediction

based

on

the

p-values

outperformed

prediction

using

other

scores

when

the

number

of

potential

domain

domain

contacts

between

two

proteins

is

relatively

high

and

hence

when

the

prediction

problem

is

harder

For

smaller

numbers

of

potential

contacts

the

p-value

method

was

not

as

successful

as

other

methods

Domain

contact

predictions

based

on

the

p-values

were

also

made

for

interacting

protein

pairs

taken

from

the

Uetz

Ito

and

MIPS

data

sets

In

this

way

we

have

suggested

novel

structural

information

for

a

large

number

of

protein

interactions

As

discussed

in

Section

the

predictive

power

of

our

method

is

limited

by

the

quality

and

coverage

of

binary

interaction

data

The

method

is

also

limited

by

the

fact

that

gaps

in

domain

assignments

are

ignored

and

by

the

assumption

that

interaction

is

mediated

by

domain

domain

contact

protein

interactions

can

also

be

mediated

by

a

domain

binding

to

a

short

protein

motif

-LRB-

Pawson

and

Nash

Puntervoll

et

al.

-RRB-

This

could

lead

to

erroneous

predictions

a

contact

could

be

predicted

between

two

domains

when

in

fact

it

is

an

adjacent

gap

that

is

mediating

the

interaction

This

gap

could

contain

another

domain

that

was

not

recognized

by

the

assignment

procedure

or

a

short

peptide

motif

of

the

type

described

above

Our

methodology

also

relies

on

the

assumption

of

a

single

domain

domain

contact

between

interacting

proteins

but

analysis

of

the

PQS

reveals

that

many

protein

interactions

involve

contacts

between

several

domains

It

is

clear

that

this

work

raises

many

questions

about

the

pattern

and

nature

of

domain

domain

contacts

in

protein

complexes

The

PQS

contains

a

wealth

of

information

about

domain

domain

contacts

in

protein

complexes

and

while

this

paper

has

examined

this

information

very

briefly

a

more

detailed

analysis

is

likely

to

be

fruitful

REFERENCES

Aloy

P.

and

Russell

R.B.

-LRB-

-RRB-

Interrogating

protein

interaction

networks

through

structural

biology

Proc

Natl

Acad

Sci

USA

Aloy

P.

Bottcher

B.

Ceulemans

H.

Leutwein

C.

Mellwig

C.

Fischer

S.

Gavin

A.C.

Bork

P.

Superti-Furga

G.

Serrano

L.

and

Russell

R.B.

-LRB-

-RRB-

Structure-based

assembly

of

protein

complexes

in

yeast

Science

Apic

G.

Gough

J.

and

Teichmann

S.A.

-LRB-

-RRB-

Domain

combinations

in

archaeal

eubacterial

and

eukaryotic

proteomes

J.

Mol

Biol.

Benjamini

Y.

and

Hochberg

Y.

-LRB-

-RRB-

Controlling

the

false

discovery

rate

a

practical

and

powerful

approach

to

multiple

testing

J.

R.

Statist

Soc

B

Deng

M.H.

Mehta

S.

Sun

F.Z.

and

Chen

T.

-LRB-

-RRB-

Inferring

domain

domain

interactions

from

protein

protein

interactions

Genome

Res.

Fitch

W.M.

-LRB-

-RRB-

Random

sequences

J.

Mol

Biol.

Gough

J.

Karplus

K.

Hughey

R.

and

Chothia

C.

-LRB-

-RRB-

Assignment

of

homology

to

genome

sequences

using

a

library

of

hidden

Markov

models

that

represent

all

proteins

of

known

structure

J.

Mol

Biol.

Henrick

K.

and

Thornton

J.M.

-LRB-

-RRB-

PQS

a

protein

quaternary

structure

file

server

Trends

Biochem

Sci.

Ito

T.

Chiba

T.

Ozawa

R.

Yoshida

M.

Hattori

M.

and

Sakaki

Y.

-LRB-

-RRB-

A

comprehensive

two-hybrid

analysis

to

explore

the

yeast

protein

interactome

Proc

Natl

Acad

Sci

USA

Kim

W.K.

Park

J.

and

Suh

J.K.

-LRB-

-RRB-

Large

scale

statistical

prediction

of

protein

protein

interaction

by

potentially

interacting

domain

pair

Genome

Inform.

Legrain

P.

Wojcik

J.

and

Gauthier

J.M.

-LRB-

-RRB-

Protein

protein

interaction

maps

a

lead

towards

cellular

functions

Trends

Genet.

Mewes

H.W.

Frishman

D.

G

ldener

U.

Mannhaupt

G.

Mayer

K.

Mokrejs

M.

Morgenstern

B.

M

nsterkoetter

M.

Rudd

S.

and

Weil

B.

-LRB-

-RRB-

MIPS

a

database

for

genomes

and

protein

sequences

Nucleic

Acids

Res.

Murzin

A.G.

Brenner

S.E.

Hubbard

T.

and

Chothia

C.

-LRB-

-RRB-

Scop

a

structural

classification

of

proteins

database

for

the

investigation

of

sequences

and

structures

J.

Mol

Biol.

Ng

S.K.

Zhang

Z.

and

Tan

S.H.

-LRB-

-RRB-

Integrative

approach

for

computationally

inferring

protein

domain

interactions

Bioinformatics

Pawson

T.

and

Nash

P.

-LRB-

-RRB-

Assembly

of

cell

regulatory

systems

through

protein

interaction

domains

Science

Puntervoll

P.

Linding

R.

Gemund

C.

Chabanis-Davidson

S.

Mattingsdal

M.

Cameron

S.

Martin

D.M.A.

Ausiello

G.

Brannetti

B.

Costantini

A.

et

al.

-LRB-

-RRB-

ELM

server

a

new

resource

for

investigating

short

functional

sites

in

modular

eukaryotic

proteins

Nucleic

Acids

Res.

Russell

R.B.

Alber

F.

Aloy

P.

Davis

F.P.

Korkin

D.

Pichaud

M.

Topf

M.

and

Sali

A.

-LRB-

-RRB-

A

structural

prespective

on

protein

protein

interactions

Curr

Opin

Struc

Biol.

Sprinzak

E.

and

Margalit

H.

-LRB-

-RRB-

Correlated

sequence-signatures

as

markers

of

protein

protein

interaction

J.

Mol

Biol.

Sprinzak

E.

Sattath

S.

and

Margalit

H.

-LRB-

-RRB-

How

reliable

are

experimental

protein

protein

interaction

data

J.

Mol

Biol.

Uetz

P.

Giot

L.

and

Cagney

G.

-LRB-

-RRB-

A

comprehensive

analysis

of

protein

protein

interactions

in

Saccharomyces

cerevisiae

Nature

von

Mering

C.

Krause

R.

Snel

B.

Cornell

M.

Oliver

S.G.

Fields

S.

and

Bork

P.

-LRB-

-RRB-

Comparative

assessment

of

large-scale

data

sets

of

protein

protein

interactions

Nature

O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B B O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O B B O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O