Inferring

Cellular

Networks

Using

Probabilistic

Graphical

Models

Nir

Friedman

et

al.

Science

-LRB-

-RRB-

DOI

science

The

following

resources

related

to

this

article

are

available

online

at

-LRB-

this

information

is

current

as

of

May

-RRB-

Updated

information

and

services

including

high-resolution

figures

can

be

found

in

the

online

version

of

this

article

at

Supporting

Online

Material

can

be

found

at

A

list

of

selected

additional

articles

on

the

Science

Web

sites

related

to

this

article

can

be

found

at

This

article

cites

articles

of

which

can

be

accessed

for

free

This

article

has

been

cited

by

article

-LRB-

s

-RRB-

on

the

ISI

Web

of

Science

This

article

has

been

cited

by

articles

hosted

by

HighWire

Press

see

This

article

appears

in

the

following

subject

collections

Computers

Mathematics

Information

about

obtaining

reprints

of

this

article

or

about

obtaining

permission

to

reproduce

this

article

in

whole

or

in

part

can

be

found

at

Science

-LRB-

print

ISSN

online

ISSN

-RRB-

is

published

weekly

except

the

last

week

in

December

by

the

American

Association

for

the

Advancement

of

Science

New

York

Avenue

NW

Washington

DC

Copyright

by

the

American

Association

for

the

Advancement

of

Science

all

rights

reserved

The

title

Science

is

a

registered

trademark

of

AAAS

Downloaded

from

on

May

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

REVIEW

Inferring

Cellular

Networks

Using

Probabilistic

Graphical

Models

Nir

Friedman

High-throughput

genome-wide

molecular

assays

which

probe

cellular

networks

from

different

perspectives

have

become

central

to

molecular

biology

Probabilistic

graphical

models

are

useful

for

extracting

meaningful

biological

insights

from

the

resulting

data

sets

These

models

provide

a

concise

representation

of

complex

cellular

networks

by

composing

simpler

submodels

Procedures

based

on

well-understood

principles

for

inferring

such

models

from

data

facilitate

a

model-based

methodology

for

analysis

and

discovery

This

methodology

and

its

capabilities

are

illustrated

by

several

recent

applications

to

gene

expression

data

Research

in

molecular

biology

is

undergoing

a

revolution

The

availability

of

complete

genome

sequences

facilitates

the

development

of

high-throughput

assays

that

can

probe

cells

at

a

genome-wide

scale

Such

assays

measure

molecular

networks

and

their

components

at

multiple

levels

These

include

mRNA

transcript

quantities

protein-protein

and

protein-DNA

interactions

chromatin

structure

and

protein

quantities

localization

and

modifications

These

rich

data

illuminate

the

working

of

cellular

processes

from

different

perspectives

and

offer

much

promise

for

novel

insights

about

these

processes

-LRB-

-RRB-

The

challenge

for

computational

biology

is

to

provide

methodologies

for

transforming

high-throughput

heterogeneous

data

sets

into

School

of

Computer

Science

and

Engineering

Hebrew

University

Jerusalem

Israel

E-mail

nir

cs.huji.ac.il

biological

insights

about

the

underlying

mechanisms

Although

high-throughput

assays

provide

a

global

picture

the

details

are

often

noisy

hence

conclusions

should

be

supported

by

several

types

of

observations

Integration

of

data

from

assays

that

examine

cellular

systems

from

different

viewpoints

-LRB-

for

instance

gene

expression

and

proteinprotein

interactions

-RRB-

can

lead

to

a

more

coherent

reconstruction

and

reduce

the

effects

of

noise

To

perform

such

an

integration

however

we

must

understand

the

biological

principles

that

couple

the

different

measurements

In

addition

the

conclusions

of

the

analysis

should

go

beyond

a

mere

description

of

the

data

and

should

provide

new

knowledge

about

the

relevant

biological

entities

and

processes

ideally

in

the

form

of

concrete

testable

hypotheses

To

answer

this

challenge

we

need

to

build

models

of

the

biological

system

A

model

is

a

simplifying

abstraction

It

gen

erates

predictions

of

system

behavior

under

different

conditions

-LRB-

as

reflected

by

observations

-RRB-

and

illuminates

the

roles

of

various

system

components

in

these

behaviors

We

focus

on

probabilistic

models

which

use

stochasticity

to

account

for

measurement

noise

variability

in

the

biological

system

and

aspects

of

the

system

that

are

not

captured

by

the

model

In

a

model-based

approach

to

data

analysis

we

start

by

defining

the

space

of

possible

models

that

we

are

willing

to

consider

This

modeling

decision

depends

on

the

phenomena

we

wish

to

describe

and

how

they

are

reflected

by

the

observations

We

then

use

a

learning

procedure

to

select

the

model

that

best

fits

the

actual

observations

-LRB-

Such

procedures

are

referred

to

by

different

names

in

different

disciplines

including

inference

estimation

reverse

engineering

and

system

identification

-RRB-

Finally

we

use

the

learned

model

to

reason

about

the

data

make

predictions

and

glean

insights

and

hypotheses

An

important

aspect

of

model-based

approaches

is

the

shift

from

a

procedural

methodology

to

a

declarative

one

In

a

procedural

method

we

focus

on

the

sequence

of

steps

from

the

data

to

the

conclusions

For

example

when

relating

transcription

factor

binding

sites

in

the

promoter

regions

SCIENCE

VOL

FEBRUARY

Downloaded

from

on

May

S.

Nee

Philos

Trans

R.

Soc

Lond

B

Biol

Sci

-LRB-

-RRB-

R.

M.

May

Stability

and

Complexity

in

Model

Ecosystems

-LRB-

Princeton

Univ.

Press

Princeton

NJ

-RRB-

B.

Sinervo

C.

M.

Lively

Nature

-LRB-

-RRB-

B.

Kerr

M.

A.

Riley

M.

W.

Feldman

B.

J.

M.

Bohannan

Nature

-LRB-

-RRB-

S.

A.

Frank

in

Foundations

of

Social

Evolution

J.

R.

Krebs

T.

H.

Clutton-Brock

Eds

-LRB-

Princeton

Univ.

Press

Princeton

NJ

-RRB-

E.

Sober

D.

S.

Wilson

Unto

Others

The

Evolution

and

Psychology

of

Unselfish

Behavior

-LRB-

Harvard

Univ.

Press

Cambridge

MA

-RRB-

M.

A.

Nowak

R.

M.

May

Nature

-LRB-

-RRB-

A.

Sasaki

W.

D.

Hamilton

F.

Ubeda

Proc

R.

Soc

London

Ser

B

-LRB-

-RRB-

P.

D.

Taylor

L.

Jonker

Math

Biosci

-LRB-

-RRB-

J.

Hofbauer

K.

Sigmund

Evolutionary

Games

and

Population

Dynamics

-LRB-

Cambridge

Univ.

Press

Cambridge

-RRB-

J.

W.

Weibull

Evolutionary

Game

Theory

-LRB-

MIT

Press

Cambridge

MA

-RRB-

C.

Hauert

S.

De

Monte

J.

Hofbauer

K.

Sigmund

Science

-LRB-

-RRB-

T.

Vincent

B.

Brown

Evolutionary

Game

Theory

Natural

Selection

and

Darwinian

Dynamics

-LRB-

Cambridge

Univ.

Press

Cambridge

-RRB-

D.

Fudenberg

K.

Levine

The

Theory

of

Learning

in

Games

-LRB-

MIT

Press

Cambridge

MA

-RRB-

R.

Ferriere

R.

Michod

Am

Nat

-LRB-

-RRB-

U.

Dieckmann

R.

Law

J.

A.

J.

Metz

Eds.

The

Geometry

of

Ecological

Interactions

Simplifying

Spatial

Complexity

-LRB-

Cambridge

Univ.

Press

Cambridge

-RRB-

R.

Durrett

SIAM

-LRB-

Soc

Ind.

Appl

Math

-RRB-

Review

-LRB-

-RRB-

R.

Cressman

Evolutionary

Dynamics

and

Extensive

Form

Games

-LRB-

MIT

Press

Cambridge

MA

-RRB-

J.

A.

J.

Metz

R.

M.

Nisbet

S.

A.

H.

Geritz

Trends

Ecol

Evol

-LRB-

-RRB-

S.

D.

Mylius

O.

J.

Diekmann

J.

Theor

Biol

-LRB-

-RRB-

R.

Boyd

J.

Lorberbaum

Nature

-LRB-

-RRB-

M.

A.

Nowak

K.

Sigmund

Acta

Appl

Math

-LRB-

-RRB-

J.

Hofbauer

K.

Sigmund

Appl

Math

Lett

-LRB-

-RRB-

U.

Dieckmann

R.

Law

J.

Math

Biol

-LRB-

-RRB-

A.

P.

Hendry

J.

K.

Wenburg

P.

Bentzen

E.

C.

Volk

T.

P.

Quinn

Science

-LRB-

-RRB-

U.

Dieckmann

P.

Marrow

R.

Law

J.

Theor

Biol

-LRB-

-RRB-

I.

Eshel

J.

Theor

Biol

-LRB-

-RRB-

F.

B.

Christiansen

Am

Nat

-LRB-

-RRB-

P.

D.

Taylor

Theor

Pop

Biol

-LRB-

-RRB-

M.

A.

Nowak

J.

Theor

Biol

-LRB-

-RRB-

J.

A.

J.

Metz

S.

A.

Geritz

G.

Meszena

F.

J.

A.

Jacobs

J.

S.

van

Heerwarden

in

Stochastic

and

Spatial

Structures

of

Dynamical

Systems

S.

J.

Van

Strien

S.

M.

Verduyn

Lunel

Eds

-LSB-

Koninklijke

Nederlandse

Academie

van

Wetenschappen

-LRB-

KNAW

-RRB-

Amsterdam

-RSB-

pp.

S.

A.

Geritz

J.

A.

J.

Metz

E.

Kisdi

G.

Meszena

Phys

Rev.

Lett

-LRB-

-RRB-

K.

Parvinen

Bull

Math

Biol

-LRB-

-RRB-

I.

Eshel

J.

Math

Biol

-LRB-

-RRB-

R.

Burger

Am

Nat

-LRB-

-RRB-

P.

Hammerstein

J.

Math

Biol

-LRB-

-RRB-

M.

Slatkin

Genetics

-LRB-

-RRB-

U.

Dieckmann

M.

Doebeli

Nature

-LRB-

-RRB-

C.

Matessi

A.

Gimelfarb

S.

Gavrilets

Select

Mol

Genes

Memes

-LRB-

-RRB-

M.

Gyllenberg

K.

Parvinen

U.

Dieckmann

J.

Math

Biol

-LRB-

-RRB-

L.

Van

Valen

Evol

Theory

-LRB-

-RRB-

For

an

interactive

Web

page

see

virtuallabs

K.

M.

Page

M.

A.

Nowak

J.

Theor

Biol

-LRB-

-RRB-

Support

from

the

Austrian

Science

Fund

WK

the

Packard

Foundation

and

J.

Epstein

is

gratefully

acknowledged

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

of

genes

to

their

expression

profiles

we

can

start

by

finding

clusters

of

coexpressed

genes

and

then

search

for

overrepresented

elements

in

the

promoters

of

the

genes

in

each

cluster

-LRB-

-RRB-

Alternatively

we

can

group

genes

with

similar

binding

sites

in

their

promoter

regions

and

then

test

whether

they

are

coexpressed

-LRB-

-RRB-

In

contrast

in

a

declarative

approach

we

start

by

designing

a

model

that

encompasses

both

gene

expression

and

binding

sites

As

explained

below

such

a

model

explicitly

describes

the

assumptions

we

make

about

the

relations

between

the

two

types

of

data

By

doing

so

we

clarify

what

kinds

of

predictions

we

can

perform

with

each

model

after

we

learn

its

parameters

from

the

data

and

how

these

parameters

relate

to

the

biological

phenomena

we

are

attempting

to

capture

Next

we

apply

well-understood

principles

such

as

maximum

likelihood

estimation

to

fit

the

model

to

the

data

This

can

be

done

using

a

general-purpose

strategy

-LRB-

such

as

Expectation

Maximization

gradient

ascent

or

Gibbs

sampling

-RRB-

in

the

context

of

the

particular

model

By

treating

different

data

sets

within

one

model

the

learning

procedure

can

combine

evidence

from

multiple

data

sets

and

reach

more

robust

conclusions

The

model-based

approach

is

widely

used

in

biological

sequence

analysis

-LRB-

-RRB-

for

which

there

is

a

range

of

established

sequence

models

such

as

Hidden

Markov

Models

For

cellular

networks

the

structure

of

the

underlying

processes

that

generate

the

observed

measurements

is

not

fully

characterized

and

the

question

of

which

mechanism

to

model

and

at

what

granularity

is

open-ended

and

depends

on

the

biological

question

we

are

attempting

to

answer

In

this

review

I

examine

the

use

of

a

class

of

mathematical

models

known

as

probabilistic

graphical

models

-LRB-

-RRB-

for

model-based

analysis

of

cellular

networks

These

models

were

developed

in

the

fields

of

machine

learning

and

statistics

for

modeling

complex

systems

with

multiple

interacting

entities

They

are

closely

related

to

probabilistic

sequence

models

but

are

not

restricted

to

sequential

observations

Probabilistic

graphical

models

are

suitable

for

this

task

for

several

reasons

They

provide

a

concise

language

for

describing

probability

distributions

over

the

observations

The

computational

procedures

for

reasoning

in

graphical

models

are

derived

from

basic

principles

of

probability

theory

In

addition

the

literature

on

graphical

models

provides

approaches

to

learning

from

data

that

are

derived

from

basic

well-understood

principles

in

statistics

These

approaches

allow

the

use

of

observations

to

fill

in

many

model

details

Furthermore

they

provide

principles

for

combining

multiple

local

models

into

a

joint

global

model

This

provides

flexibility

when

construct

ing

models

for

novel

data

sets

or

experimental

designs

Using

graphical

models

one

can

construct

simple

submodels

and

then

combine

them

for

the

full

model

Finally

the

declarative

nature

of

graphical

models

provides

an

advantage

when

we

need

to

extend

the

model

to

account

for

additional

aspects

of

the

system

or

new

observations

My

emphasis

is

on

the

modeling

choices

and

how

they

facilitate

different

analysis

tasks

For

each

of

these

models

we

also

need

to

apply

inference

and

learning

procedures

In

some

cases

we

can

adopt

generic

strategies

In

others

finding

computationally

efficient

algorithms

is

a

major

challenge

and

the

details

of

the

algorithms

-LRB-

not

discussed

here

-RRB-

greatly

affect

the

results

Probabilistic

Graphical

Models

When

modeling

a

biological

system

we

are

interested

in

entities

that

are

involved

in

the

system

under

study

-LRB-

e.g.

genes

-RRB-

and

their

different

attributes

-LRB-

e.g.

expression

level

-RRB-

In

a

probabilistic

model

we

treat

each

of

these

attributes

as

a

random

variable

-LRB-

-RRB-

Random

variables

include

observed

attributes

such

as

the

expression

level

of

a

particular

gene

in

a

particular

experiment

as

well

as

hidden

attributes

that

are

assumed

by

the

model

such

as

the

cluster

assignment

of

a

particular

gene

A

model

embodies

the

description

of

the

joint

probability

distribution

of

all

the

random

variables

of

interest

Probabilistic

graphical

models

represent

multivariate

joint

probability

distributions

via

a

product

of

terms

each

of

which

involves

only

a

few

variables

The

structure

of

the

product

is

represented

by

a

graph

that

relates

variables

that

appear

in

a

common

term

This

graph

specifies

the

product

form

of

the

distribution

and

also

provides

tools

for

reasoning

about

the

properties

entailed

by

the

product

-LRB-

-RRB-

For

a

sparse

graph

the

representation

is

compact

and

in

many

cases

allows

effective

inference

and

learning

In

Bayesian

Networks

the

joint

distribution

over

a

set

X

-LCB-

Xn

-RCB-

of

random

variables

is

represented

as

a

product

of

conditional

probabilities

A

Bayesian

network

associates

with

each

variable

Xi

a

conditional

probability

P

-LRB-

Xi

Ui

-RRB-

where

Ui

X

is

the

set

of

variables

that

are

called

the

parents

of

Xi

Intuitively

the

values

of

the

parents

directly

influence

the

choice

of

value

for

Xi

The

resulting

product

is

of

the

form

P

-LRB-

X

X

n

-RRB-

i

P

-LRB-

X

X

n

-RRB-

Z

j

j

-LSB-

C

j

-RSB-

-LRB-

-RRB-

P

-LRB-

X

i

U

i

-RRB-

-LRB-

-RRB-

The

graphical

representation

is

given

by

a

directed

graph

where

we

put

edges

from

Xi

's

parents

to

Xi

-LRB-

Fig.

A

to

C

-RRB-

If

the

graph

is

acyclic

the

product

decomposition

of

Eq

is

guaranteed

to

be

a

coherent

probability

distribution

where

j

-LSB-

Cj

-RSB-

is

the

jth

potential

over

the

X

and

Z

is

a

normalizing

variables

Cj

constant

that

ensures

that

the

total

probability

mass

is

-LRB-

Fig.

D

to

F

-RRB-

A

canonical

example

of

an

undirected

model

is

an

Ising

model

where

each

random

variable

represents

the

orientation

of

an

element

-LRB-

e.g.

magnetic

particle

-RRB-

and

the

potential

between

pairs

of

elements

captures

the

compatibility

of

two

elements

The

joint

probability

is

determined

by

the

overall

compatibility

of

each

assignment

of

values

according

to

all

the

potentials

Another

related

class

of

models

are

Chain

Graphs

which

involve

a

product

of

conditional

probabilities

and

potentials

In

many

domains

there

is

additional

structure

beyond

the

product

form

that

can

be

exploited

for

concise

representation

-LRB-

-RRB-

Below

we

discuss

one

such

class

of

models

that

captures

additional

structure

A

crucial

question

for

the

tasks

we

examine

here

is

inferring

or

learning

models

from

observations

-LRB-

-RRB-

The

general

aim

is

to

learn

a

model

that

is

as

close

as

possible

to

the

underlying

distribution

from

which

the

observations

were

made

We

distinguish

two

main

tasks

parameter

estimation

and

model

selection

In

parameter

estimation

we

learn

the

parameters

of

the

conditional

probabilities

for

a

given

model

structure

This

task

is

often

addressed

as

a

FEBRUARY

VOL

SCIENCE

Downloaded

from

on

May

Bayesian

networks

appear

naturally

in

several

domains

in

biology

In

pedigree

analysis

for

example

the

joint

distribution

of

genotypes

in

a

pedigree

is

a

product

of

conditional

probabilities

of

the

genotype

of

each

individual

given

the

genotypes

of

its

two

biological

parents

In

phylogenetic

models

the

probability

over

all

possible

sequences

during

evolution

is

the

product

of

the

conditional

probability

of

each

sequence

given

its

latest

ancestral

sequence

in

the

phylogeny

To

specify

a

model

completely

we

need

to

describe

the

conditional

probability

associated

with

each

variable

In

general

any

statistical

regression

model

may

be

used

For

example

we

can

consider

models

where

each

P

-LRB-

Xi

Ui

-RRB-

is

a

linear

regression

of

Xi

on

Ui

Alternatively

we

can

use

decision

trees

to

represent

the

probability

of

a

discrete

variable

Xi

given

the

values

of

its

parents

The

choice

of

a

specific

parametric

representation

of

the

conditional

probabilities

is

often

dictated

by

our

knowledge

or

assumptions

about

the

domain

Another

class

of

models

are

Markov

Networks

which

represent

a

joint

distribution

as

a

product

of

potentials

Each

potential

captures

the

interactions

among

a

-LRB-

small

-RRB-

set

of

variables

and

specifies

the

desirability

of

joint

value

assignments

to

these

variables

This

results

in

a

product

of

the

form

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

Bayesian

network

once

we

are

given

the

set

of

genes

and

arrays

The

model

just

described

can

achieve

high

likelihood

if

the

cluster

and

gene

assignment

partitions

the

original

measurements

into

blocks

with

approximately

uniform

expression

within

each

block

-LRB-

-RRB-

We

can

learn

such

a

partition

by

using

an

Expectation

Maximization

procedure

that

iterates

between

an

E-step

which

uses

current

parameters

to

find

the

probabilistic

cluster

assignment

of

genes

and

arrays

and

an

M-step

which

reestimates

the

distribution

within

each

gene

array

cluster

combination

on

the

basis

of

this

assignment

This

basic

model

can

be

extended

to

capture

additional

insight

about

the

biological

mechanisms

We

consider

one

example

here

It

From

Clustering

to

is

common

to

assume

that

coexRegulation

pression

of

genes

reflects

coregulaWe

now

consider

a

graphical

modtion

A

key

regulation

mechanism

el

for

gene

expression

data

and

involves

binding

of

transcription

then

examine

how

to

extend

it

to

factors

to

promoter

regions

of

the

modeling

of

cis-regulatory

elegenes

Thus

we

aim

to

identify

the

ments

The

main

sources

of

hightranscription

factor

binding

sites

in

throughput

data

on

the

behavior

of

the

promoter

region

of

genes

that

cellular

networks

are

gene

exprescan

explain

observed

coexpression

sion

profiles

obtained

using

DNA

To

do

this

we

extend

the

model

to

microarrays

A

typical

data

set

reinclude

observations

about

promotports

the

expression

level

of

thouer

sequences

A

straightforward

apsands

of

genes

as

measured

by

sevproach

is

to

maintain

the

clustering

eral

dozens

or

hundreds

of

arrays

model

where

genes

in

the

same

clusWe

treat

these

as

observations

of

ter

have

similar

expression

patterns

the

values

of

random

variables

and

in

addition

associate

each

cluster

Xg

a

where

g

is

an

index

over

with

the

transcription

factors

that

reggenes

and

a

is

an

index

over

arrays

ulate

it

Although

such

a

description

A

fairly

simple

modeling

asoversimplifies

the

biological

mechasumption

is

that

genes

can

be

parnism

it

can

capture

the

first-order

titioned

into

clusters

of

coexsignal

while

ignoring

many

differpressed

genes

and

that

the

genes

ences

between

the

expression

and

in

each

cluster

have

a

typical

exregulation

of

specific

genes

pression

level

in

each

array

We

There

are

several

ways

to

might

also

assume

that

arrays

are

Fig.

-LRB-

A

-RRB-

A

Bayesian

network

over

five

binary

random

variables

Vertices

translate

this

intuition

into

a

partitioned

into

array

clusters

are

labeled

with

random

variable

names

-LRB-

A

to

E

-RRB-

edges

correspond

to

mathematical

model

One

apdirect

dependencies

-LRB-

B

-RRB-

The

product

form

specified

by

this

Bayesian

which

capture

relevant

biological

network

structure

A

full

specification

of

the

joint

distribution

for

these

proach

is

to

annotate

promoters

context

and

that

the

expression

random

variables

requires

parameters

this

product

form

requires

with

characterized

binding

sites

of

a

gene

is

roughly

the

same

in

parameters

-LRB-

C

-RRB-

An

example

of

one

conditional

distribution

in

the

product

and

then

use

these

as

new

atarrays

that

belong

to

the

same

form

that

specifies

P

-LRB-

C

A

B

-RRB-

-LRB-

D

-RRB-

A

Markov

network

over

the

same

five

tributes

of

the

gene

entity

The

array

cluster

We

can

pose

this

variables

-LRB-

E

-RRB-

A

product

form

that

induces

this

Markov

network

structure

random

binary

variable

Rg

j

demodel

by

adding

random

vari

This

is

a

product

of

four

potential

functions

each

a

function

of

a

subset

notes

whether

gene

g

has

a

bindof

the

variables

-LRB-

F

-RRB-

One

potential

in

this

product

form

ing

site

of

transcription

factor

j.

ables

so

that

GeneClusterg

denotes

the

cluster

assignment

of

Then

we

can

design

a

model

gene

g

and

ArrayClustera

denotes

the

cluster

where

the

cluster

assignment

of

each

gene

scheme

or

template

is

shared

by

all

entities

assignment

of

array

a

The

underlying

asdirectly

influences

the

associated

binding

of

the

same

type

For

example

the

condisumption

is

that

the

expression

of

gene

g

in

sites

and

expression

attributes

-LRB-

-RRB-

tional

probability

of

P

-LRB-

Xg

a

GeneClusterg

ArrayCluster

a

-RRB-

is

similar

for

different

array

a

depends

on

the

value

of

GeneClusterg

Alternatively

we

can

attempt

to

maxichoices

of

g

and

a.

By

capturing

such

and

ArrayClustera

This

model

assumes

that

mize

the

likelihood

of

gene

expression

proregularities

we

can

provide

a

more

concise

all

measurements

that

correspond

to

a

particfiles

given

the

promoter

region

content

representation

of

the

model

One

such

repular

gene

cluster

array

cluster

pair

are

govAgain

we

introduce

binding

site

attributes

to

resentation

is

the

language

of

relational

erned

by

the

same

conditional

distribution

the

gene

entity

but

now

we

assume

that

the

Bayesian

networks

-LRB-

-RRB-

Figure

We

can

describe

such

a

model

as

a

gene

cluster

depends

on

these

attributes

-LRB-

shows

a

template

model

for

the

clustering

Bayesian

network

The

structure

of

the

-RRB-

Such

a

model

focuses

on

binding

sites

problem

from

which

we

can

generate

the

Bayesian

network

is

regular

in

the

sense

that

predict

expression

and

does

not

attempt

SCIENCE

VOL

FEBRUARY

maximum

likelihood

problem

In

model

selection

we

select

among

different

model

structures

to

find

one

that

best

reflects

the

dependencies

in

the

domain

This

task

is

often

addressed

as

a

discrete

optimization

problem

where

we

try

to

maximize

a

score

that

measures

how

well

each

candidate

structure

matches

the

observed

data

-LRB-

-RRB-

Once

we

specify

or

learn

a

model

that

describes

a

joint

distribution

we

can

use

it

to

compute

the

likelihood

of

our

observations

and

make

predictions

about

the

value

of

unobserved

random

variables

given

these

observations

There

is

a

wide

choice

of

exact

and

approximate

methods

for

answering

such

queries

These

exploit

when

possible

the

structure

of

the

product

form

for

efficient

computation

-LRB-

-RRB-

that

each

expression

attribute

Xg

a

has

parents

GeneClusterg

and

ArrayClustera

The

actual

network

structure

depends

on

the

number

of

genes

and

arrays

in

the

data

set

-LRB-

see

Fig.

for

a

small

example

-RRB-

The

description

by

a

Bayesian

network

however

does

not

explicitly

represent

two

important

aspects

First

the

random

variables

denote

attributes

of

different

entities

such

as

genes

and

arrays

Second

a

general

Downloaded

from

on

May

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

to

account

for

the

occurrences

of

other

binding

sites

which

might

be

relevant

under

conditions

that

are

not

represented

in

the

expres

sion

data

We

can

augment

this

model

by

modeling

the

probability

of

a

binding

site

given

the

actual

promoter

sequence

We

in

Fig.

-LRB-

A

-RRB-

A

Bayesian

network

for

the

clustering

problem

in

a

simple

data

set

with

three

genes

and

two

arrays

The

random

variable

Xg

a

denotes

the

spot

that

measures

expression

of

gene

g

in

array

a

GCg

denotes

the

cluster

of

gene

g

and

ACa

denotes

the

array

cluster

of

array

a.

-LRB-

B

-RRB-

A

relational

Bayesian

network

template

for

the

basic

clustering

model

Boxes

delimit

entity

types

dotted

lines

correspond

to

relations

between

entities

-LRB-

e.g.

the

relation

between

a

spot

entity

and

a

gene

entity

denotes

that

the

spot

measures

the

expression

of

the

particular

gene

-RRB-

Each

spot

entity

has

a

single

attribute

that

measures

an

expression

level

and

is

associated

with

a

gene

entity

and

an

array

entity

Each

gene

entity

is

associated

with

a

gene

cluster

and

each

array

entity

with

an

array

cluster

-LRB-

C

-RRB-

Representation

of

the

conditional

distribution

of

expression

levels

given

the

clusters

of

the

corresponding

gene

and

array

Each

combination

of

values

of

the

respective

gene-cluster

and

array-cluster

variables

is

associated

with

parameters

of

a

Gaussian

distribution

-LRB-

mean

and

standard

deviation

-RRB-

-LRB-

D

-RRB-

A

template

model

that

also

includes

promoter

sequences

Each

gene

entity

is

associated

with

a

promoter

entity

that

has

a

sequence

attribute

that

reports

the

DNA

sequence

of

the

promoter

The

gene

entity

now

has

new

attributes

where

Rj

indicates

whether

the

gene

is

regulated

by

the

jth

transcription

factor

This

indicator

depends

on

the

promoter

sequence

The

cluster

of

the

gene

depends

on

the

combination

of

these

indicators

troduce

new

entities

that

denote

the

promoter

regions

and

model

Rg

j

as

depending

on

the

promoter

sequence

Seqg

-LRB-

Fig.

-RRB-

The

parameters

of

this

conditional

probability

characterize

the

specific

motif

recognized

by

the

transcription

factor

This

extension

allows

us

to

learn

the

characterization

of

the

binding

site

while

learning

how

its

presence

influences

gene

expression

A

crucial

detail

in

building

such

a

model

is

the

representation

of

the

conditional

distributions

associated

with

GeneClusterg

This

distribution

describes

how

the

existence

of

binding

sites

in

the

promoter

region

determines

-LRB-

or

predicts

-RRB-

what

cluster

the

gene

belongs

to

The

conditional

probabilities

explored

so

far

involve

fairly

generic

representation

of

decision

trees

-LRB-

-RRB-

or

additive

votes

-LRB-

-RRB-

Both

representations

manage

to

reconstruct

some

aspects

of

yeast

transcriptional

circuits

However

it

is

not

clear

whether

either

one

matches

the

underlying

logic

in

biological

regulation

Learning

in

this

type

of

model

combines

similarity

of

genes

in

terms

of

expression

and

in

terms

of

their

promoters

In

training

the

model

there

are

steps

where

we

find

new

binding

sites

that

explain

assignments

of

genes

to

clusters

steps

that

reassign

genes

to

new

clusters

on

the

basis

of

both

their

expression

profile

and

their

promoter

region

and

steps

that

reestimate

the

distributions

of

expression

within

each

cluster

Thus

learning

involves

information

flow

between

the

two

types

of

data

and

allows

the

combination

of

weak

evidence

from

both

sources

This

information

is

channeled

through

the

gene

cluster

variables

Thus

to

achieve

high

likelihood

the

gene

cluster

variables

must

represent

clusters

of

genes

with

both

coherent

expression

profiles

and

similar

promoters

At

the

same

time

the

learning

procedure

must

identify

the

binding

site

motifs

that

are

most

predictive

of

these

cluster

assignments

Segal

et

al.

-LRB-

-RRB-

applied

such

a

model

to

two

data

sets

of

yeast

gene

expression

profiles

One

involved

genes

in

arrays

of

different

yeast

cell

cycle

stages

-LRB-

-RRB-

the

other

involved

genes

in

arrays

of

yeast

under

environmental

stress

conditions

-LRB-

-RRB-

They

showed

that

without

using

prior

knowledge

their

procedure

identified

several

dozen

binding

site

motifs

More

than

half

of

these

corresponded

to

motifs

that

have

been

characterized

in

the

literature

the

resulting

gene

clusters

corresponded

to

known

biological

processes

and

function

annotations

This

correspondence

is

significantly

more

pronounced

than

for

standard

clusters

learned

from

gene

expression

alone

In

addition

their

model

suggests

a

specific

cis-regulatory

circuit

that

in

many

cases

corresponds

to

prior

knowledge

about

regulation

in

yeast

The

clustering

model

can

be

extended

to

involve

other

types

of

mechanisms

and

ob

FEBRUARY

VOL

SCIENCE

Downloaded

from

on

May

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

Reconstruction

of

Regulatory

Networks

A

key

challenge

in

gene

expression

analysis

is

the

reconstruction

of

regulatory

networks

A

simple

thought

experiment

suggests

the

following

formulation

If

the

expression

of

gene

A

is

regulated

by

proteins

B

and

C

then

A

's

expression

level

is

a

function

of

the

joint

activity

levels

of

B

and

C.

Because

of

variability

in

underlying

biology

and

measurement

noise

we

treat

the

expression

of

A

as

a

stochastic

function

of

its

regulators

This

suggests

a

Bayesian

network

where

the

expression

level

of

each

gene

depends

on

the

activity

levels

of

its

regulators

In

most

current

biological

data

sets

however

we

do

not

have

access

to

measurements

of

protein

activity

levels

Hence

we

resort

to

using

expression

levels

of

genes

as

a

proxy

for

the

activity

level

of

the

proteins

they

encode

This

is

a

problematic

assumption

as

there

are

numerous

examples

where

an

activation

or

silencing

of

a

regulator

is

carried

out

by

posttranscriptional

protein

modifications

With

this

caveat

in

mind

we

set

out

to

find

a

Bayesian

network

that

relates

the

expression

level

of

a

gene

to

those

of

its

regulators

-LRB-

-RRB-

That

is

we

search

for

a

Bayesian

network

that

specifies

for

each

gene

g

a

set

of

regulators

so

that

in

each

array

Xg

a

depends

on

the

expression

level

of

the

regulators

in

that

array

We

then

use

tools

for

structure

learning

in

Bayesian

networks

-LRB-

-RRB-

to

determine

the

network

architecture

This

involves

considering

different

network

structures

and

evaluating

the

likelihood

that

they

have

generated

the

observations

This

general

outline

faces

two

main

challenges

The

first

challenge

involves

statistical

robustness

Building

a

network

that

involves

thousands

of

genes

from

several

dozen

examples

of

their

joint

expression

levels

-LRB-

arrays

-RRB-

is

extremely

problematic

Such

a

small

number

of

examples

does

not

suffice

to

distinguish

SCIENCE

VOL

FEBRUARY

Downloaded

from

on

May

servations

For

example

we

can

assume

that

active

transcription

factor

binding

sites

should

correspond

to

observations

of

transcription

factor

location

data

-LRB-

-RRB-

We

can

extend

the

model

to

view

these

observations

as

a

noisy

sensor

-LRB-

-RRB-

Another

orthogonal

extension

can

incorporate

protein-protein

interactions

For

example

a

pair

of

interacting

proteins

are

more

likely

to

belong

to

the

same

coregulated

cluster

-LRB-

because

they

might

act

together

-RRB-

To

capture

this

we

can

assign

a

Markov

network

pairwise

potential

that

prefers

coordinated

cluster

assignments

for

pairs

of

interacting

proteins

-LRB-

-RRB-

This

model

can

combine

annotation

of

proteins

-LRB-

e.g.

cluster

assignment

-RRB-

with

protein-protein

interaction

data

This

pairwise

interaction

model

can

be

applied

to

other

types

of

protein

annotations

Deng

et

al.

-LRB-

-RRB-

recently

used

a

similar

model

for

predicting

functional

annotations

of

proteins

between

true

correlations

and

spurious

ones

There

are

several

strategies

to

deal

with

this

challenge

Methods

such

as

the

bootstrap

can

identify

significant

network

features

that

are

robust

to

perturbations

of

the

observations

-LRB-

-RRB-

Another

approach

is

to

use

prior

knowledge

about

biological

principles

to

restrict

the

set

of

network

structures

we

are

willing

to

consider

-LRB-

-RRB-

This

reduces

the

number

of

competing

false

structures

and

increases

robustness

Alternatively

we

can

restrict

ourselves

to

evaluating

a

much

smaller

set

of

structures

on

the

basis

of

prior

biological

knowledge

about

specific

genes

-LRB-

-RRB-

Finally

we

can

rely

on

biological

principles

for

restricting

the

stochastic

function

of

a

gene

given

its

regulators

to

be

of

a

particular

form

-LRB-

-RRB-

The

second

and

more

difficult

challenge

is

the

biological

interpretability

of

the

results

Can

we

really

distinguish

regulation

from

coexpression

Do

these

methods

discover

direct

or

indirect

regulation

How

do

unobserved

posttranscriptional

events

affect

the

conclusions

Whereas

our

ultimate

goal

is

to

identify

the

direct

regulation

of

targets

by

transcription

factors

experience

shows

that

the

methods

also

find

many

other

indirect

relations

As

a

specific

example

we

applied

-LRB-

-RRB-

the

bootstrap

procedure

to

a

data

set

of

the

expression

profiles

of

genes

in

knockout

variants

of

yeast

-LRB-

-RRB-

This

was

done

in

an

ab

initio

fashion

without

any

prior

knowledge

about

which

genes

might

be

regulators

and

without

making

strong

assumptions

about

the

network

structure

-LRB-

Fig.

-RRB-

We

used

the

bootstrap

procedure

to

assign

confidence

to

different

relations

in

the

learned

networks

and

compiled

subnetworks

of

genes

that

have

high-confidence

interconnections

among

them

-LRB-

Fig.

-RRB-

We

then

compared

the

significant

regulation

relations

to

the

experimental

literature

This

analysis

showed

that

many

regulator-target

pairs

correspond

to

known

regulatory

relationships

that

involve

some

intermediate

steps

for

example

components

of

mitogen-activated

protein

kinase

signaling

cascades

and

their

downstream

transcriptional

targets

Two

studies

have

used

this

insight

as

a

justification

to

focus

on

regulators

that

include

components

of

signal

pathways

receptors

and

transcription

factors

Pe'er

et

al.

-LRB-

-RRB-

introduced

a

method

that

examines

only

those

networks

in

which

a

small

number

of

regulators

explain

the

expression

of

all

other

genes

-LRB-

Fig.

-RRB-

The

restriction

to

such

a

network

architecture

forces

the

learning

procedure

to

identify

the

most

pronounced

regulators

in

the

data

set

It

also

simplifies

the

learning

procedure

which

leads

to

statistical

and

computational

advantages

They

applied

this

method

to

several

yeast

gene

expression

data

sets

-LRB-

-RRB-

and

then

performed

a

systematic

validation

of

the

regulators

in

the

learned

networks

by

examining

the

process

and

function

annotation

of

the

target

set

of

each

regulator

-LRB-

Fig.

-RRB-

In

most

cases

the

significant

annotations

matched

the

known

literature

about

the

regulators

Segal

et

al.

-LRB-

-RRB-

introduced

a

method

that

examines

networks

composed

of

modules

of

coregulated

genes

-LRB-

Fig.

-RRB-

In

these

networks

all

the

genes

within

a

single

module

are

controlled

by

a

shared

regulatory

program

The

learning

procedure

simultaneously

identifies

the

composition

of

the

modules

and

finds

regulators

for

each

module

that

best

predict

the

expression

of

its

genes

The

module

network

representation

forces

the

learning

procedure

to

find

patterns

that

are

shared

by

many

genes

Such

a

representation

is

congruent

with

biological

principles

that

suggest

that

the

cellular

regulatory

circuits

coordinate

activation

or

repression

of

groups

of

genes

that

are

involved

in

the

same

process

This

is

in

contrast

to

the

two

approaches

we

described

above

where

each

gene

is

associated

with

an

individual

regulatory

program

The

benefit

of

the

module

network

approach

is

that

shared

regulation

programs

require

far

fewer

parameters

and

increase

the

robustness

of

the

learned

model

The

downside

is

that

we

lose

flexibility

and

might

miss

fine

points

in

the

regulation

of

specific

genes

Another

important

aspect

of

the

module

networks

is

that

the

representation

leads

to

an

easier

interpretation

Instead

of

examining

the

regulation

program

of

hundreds

of

genes

we

focus

on

a

much

smaller

number

of

modules

This

approach

was

applied

to

the

genes

in

arrays

of

yeast

under

environmental

stress

conditions

-LRB-

-RRB-

to

learn

a

network

with

modules

-LRB-

-RRB-

This

was

followed

by

a

systematic

evaluation

against

the

literature

which

showed

that

regulators

of

modules

agree

with

experimental

results

in

the

literature

and

with

evidence

based

on

gene

annotation

and

cis-regulatory

motifs

-LRB-

Fig.

-RRB-

Moreover

the

model

learned

by

this

method

leads

to

testable

hypotheses

of

the

form

protein

X

regulates

a

module

of

genes

G

under

condition

C.

Segal

et

al.

confirmed

such

hypotheses

for

three

unknown

regulators

-LRB-

one

transcription

factor

and

two

kinases

-RRB-

by

examining

gene

expression

of

knockout

strains

in

the

specific

conditions

where

the

regulator

is

predicted

to

be

active

These

results

show

that

regulation

can

be

learned

from

expression

profiles

Clearly

because

these

methods

examine

expression

profiles

they

detect

coordinated

changes

in

transcript

levels

of

the

target

genes

and

their

regulators

This

suggests

that

to

understand

the

successes

of

such

methods

we

need

to

examine

how

the

discovered

regulators

are

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

themselves

transcriptionally

regulated

These

regulation

mechanisms

often

involve

feedforward

loops

and

feedback

mechanisms

-LRB-

-RRB-

that

change

the

mRNA

expression

level

of

regulators

when

their

protein

activity

changes

As

a

consequence

we

can

detect

coordinated

changes

in

the

expression

levels

of

regulators

and

their

targets

This

hypothesis

is

supported

by

an

analysis

of

the

discovered

regulatory

relations

against

a

database

of

protein-DNA

and

proteinprotein

interactions

-LRB-

-RRB-

Fig.

Different

regulatory

network

architectures

-LRB-

A

-RRB-

An

unconstrained

acyclic

network

where

each

gene

can

have

a

different

regulator

set

This

is

a

fragment

of

a

network

learned

in

the

experiments

of

Pe'er

et

al.

-LRB-

-RRB-

-LRB-

B

-RRB-

A

summary

of

direct

neighbor

relations

among

the

genes

shown

in

-LRB-

A

-RRB-

based

on

bootstrap

estimates

Degrees

of

confidence

are

denoted

by

edge

thickness

We

automatically

identify

a

subnetwork

of

genes

with

high-confidence

relations

among

them

that

are

involved

in

the

yeast-mating

pathways

The

colors

highlight

genes

with

known

function

in

mating

including

signal

transduction

-LRB-

yellow

-RRB-

transcription

factors

-LRB-

blue

-RRB-

and

downstream

effectors

-LRB-

green

-RRB-

-LRB-

C

-RRB-

A

fragment

of

a

two-level

network

described

by

Pe'er

et

al.

-LRB-

-RRB-

The

top

level

contains

a

small

number

of

regulators

the

bottom

level

contains

all

other

genes

-LRB-

targets

-RRB-

Each

gene

has

different

regulators

from

among

the

regulator

genes

-LRB-

D

-RRB-

Visualization

of

significant

Gene

Ontology

-LRB-

-RRB-

annotations

of

the

targets

of

different

regulators

Each

significant

annotation

for

the

targets

of

a

regulator

-LRB-

or

pairs

of

regulators

-RRB-

is

shown

with

the

hypergeometric

p-value

-LRB-

E

-RRB-

A

fragment

of

the

module

network

described

by

Segal

et

al.

-LRB-

-RRB-

Each

module

contains

several

genes

that

share

the

same

set

of

regulators

and

share

the

same

conditional

regulation

program

given

these

regulators

-LRB-

F

-RRB-

Visualization

of

the

expression

levels

of

the

genes

in

Module

-LRB-

b

-RRB-

and

their

regulators

-LRB-

a

-RRB-

Significant

Gene

Ontology

annotations

-LRB-

c

-RRB-

and

cisregulatory

motifs

in

promoter

regions

of

genes

in

the

module

-LRB-

d

-RRB-

are

shown

-LSB-

See

figure

of

-LRB-

-RRB-

reproduced

with

permission

-RSB-

FEBRUARY

VOL

SCIENCE

Downloaded

from

on

May

MATHEMATICS

IN

BIOLOGY

SPECIAL

SECTION

Conclusion

We

have

discussed

several

model-based

approaches

for

learning

cellular

networks

from

data

These

approaches

represent

the

state

of

the

art

for

this

task

and

were

evaluated

through

extensive

validation

against

prior

biological

knowledge

and

independent

experimental

assays

A

recurring

theme

in

these

approaches

is

an

exploration

of

the

tradeoff

between

two

contradictory

aims

On

one

hand

we

aim

to

reconstruct

detailed

models

on

the

other

hand

we

must

be

able

to

learn

these

models

from

the

available

data

As

a

result

of

this

tradeoff

the

methods

we

discussed

capture

only

the

aspects

that

were

deemed

most

crucial

to

the

biological

question

at

hand

There

are

two

main

strategies

for

obtaining

models

that

provide

deeper

biological

insight

Much

current

research

focuses

on

unified

models

that

combine

evidence

from

different

levels

of

the

cellular

machinery

A

key

strength

of

the

graphical

models

is

the

ability

to

specify

models

that

account

for

such

heterogeneous

observations

These

models

can

reach

conclusions

that

are

not

supported

by

either

data

source

considered

independently

We

discussed

a

unified

model

that

finds

transcription

factor

binding

sites

and

simultaneously

characterizes

the

behavior

of

their

target

genes

Similarly

we

expect

that

modeling

promoter

sequences

will

lead

to

more

accurate

regulatory

models

-LSB-

e.g.

-LRB-

-RRB-

-RSB-

Another

direction

is

to

incorporate

data

from

assays

in

proteomics

For

example

recent

works

-LRB-

-RRB-

combine

protein-DNA

and

proteinprotein

interaction

maps

to

reconstruct

reg

E.

Lander

Nature

Genet

-LRB-

-RRB-

S.

Tavazoie

J.

D.

Hughes

M.

J.

Campbell

R.

J.

Cho

G.

M.

Church

Nature

Genet

-LRB-

-RRB-

Y.

Pilpel

P.

Sudarsanam

G.

Church

Nature

Genet

-LRB-

-RRB-

R.

Durbin

S.

Eddy

A.

Krogh

G.

Mitchison

Biological

Sequence

Analysis

Probabilistic

Models

of

Proteins

and

Nucleic

Acids

-LRB-

Cambridge

Univ.

Press

Cambridge

-RRB-

J.

Pearl

Probabilistic

Reasoning

in

Intelligent

Systems

-LRB-

Morgan

Kaufmann

San

Francisco

-RRB-

M.

I.

Jordan

Ed.

Learning

in

Graphical

Models

-LRB-

MIT

Press

Cambridge

MA

-RRB-

References

and

Notes

Supporting

Online

Material

Glossary

SCIENCE

VOL

FEBRUARY

Downloaded

from

on

May

One

of

the

key

challenges

is

to

distinguish

regulation

from

coexpression

We

can

improve

these

distinctions

by

careful

experimental

design

and

by

combining

additional

data

sources

An

important

strategy

for

dissecting

the

direction

of

regulation

is

by

gene

knockout

perturbations

The

intuition

is

that

by

knocking

out

a

gene

we

can

pinpoint

genes

downstream

from

it

The

Bayesian

network

semantics

can

use

such

perturbations

to

also

infer

the

direction

of

regulation

in

genes

upstream

of

the

knockout

-LRB-

-RRB-

and

modeling

such

perturbations

can

lead

to

stronger

conclusions

-LRB-

-RRB-

An

alternative

experimental

strategy

for

distinguishing

correlation

from

causation

is

to

examine

how

the

system

changes

over

time

-LRB-

-RRB-

For

example

Kim

et

al.

-LRB-

-RRB-

showed

that

a

learning

temporal

model

of

yeast

cell

cycle

expression

data

could

reduce

the

errors

made

by

a

model

that

did

not

take

temporal

observations

into

account

Finally

we

can

bias

the

model

to

prefer

regulator-target

pairs

that

are

consistent

with

additional

data

sources

such

as

transcription

factor

location

data

-LRB-

-RRB-

ulatory

circuits

that

explain

differential

expression

in

gene

knockout

experiments

Another

key

ingredient

for

improving

our

models

is

our

current

understanding

of

the

biological

regulatory

mechanisms

As

we

have

seen

ab

initio

exploration

can

lead

to

insights

about

the

nature

of

the

data

such

as

the

ability

to

infer

indirect

regulation

However

when

possible

incorporating

biological

principles

into

the

design

of

the

models

-LRB-

e.g.

constraining

regulatory

network

structure

-RRB-

can

restrict

the

degrees

of

freedom

during

learning

and

result

in

better

models

One

of

the

most

intriguing

prospects

for

research

is

to

develop

models

that

capture

aspects

of

the

actual

details

of

the

regulatory

machinery

This

includes

combining

tools

from

a

large

body

of

research

on

kinematic

equations

and

stochastic

differential

equations

for

modeling

cellular

pathways

-LRB-

-RRB-

In

the

near

future

we

expect

to

see

an

explosion

in

the

quantity

and

diversity

of

high-throughput

data

sets

including

new

experimental

assays

new

experimental

designs

and

examinations

of

systems

at

the

levels

of

a

single

cell

a

composite

organ

a

whole

organism

and

a

society

Computational

analysis

methods

will

be

critical

for

gleaning

biological

insight

from

these

data

sets

It

is

clear

that

no

single

tool

will

meet

all

the

analysis

needs

instead

we

need

a

range

of

tools

tailored

to

specific

assays

and

experimental

designs

To

cope

with

these

challenges

the

field

of

computational

biology

must

develop

methodologies

and

concrete

implementations

that

empower

researchers

to

explore

different

models

of

varying

detail

and

rapidly

apply

them

to

diverse

data

sets

The

language

of

graphical

models

is

well

suited

for

composing

different

submodels

in

a

principled

and

understandable

fashion

The

declarative

semantics

of

graphical

models

provide

foundations

for

building

a

modeling

toolbox

and

unified

learning

algorithms

that

apply

wellunderstood

statistical

principles

Such

tools

will

enable

researchers

to

combine

components

and

to

tailor

learning

procedures

The

challenge

is

to

gain

an

understanding

of

the

modeling

choices

for

different

cellular

components

and

their

suitability

for

different

types

of

assays

and

to

extend

the

methods

for

inference

and

learning

in

such

models

A

glossary

of

this

and

other

technical

terms

appears

on

Science

Online

W.

Buntine

J.

Artif

Intel

Res

-LRB-

-RRB-

F.

V.

Jensen

An

Introduction

to

Bayesian

Networks

-LRB-

Univ.

College

London

Press

London

-RRB-

N.

Friedman

L.

Getoor

D.

Koller

A.

Pfeffer

Relational

Data

Mining

S.

Dvzeroski

N.

Lavrac

Eds

-LRB-

SpringerVerlag

Berlin

-RRB-

pp.

E.

Segal

B.

Taskar

A.

Gasch

N.

Friedman

D.

Koller

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

Y.

Barash

N.

Friedman

J.

Comp

Biol

-LRB-

-RRB-

I.

Holmes

W.

Bruno

Proc

Int

Conf

Intell

Syst

Mol

Biol

-LRB-

-RRB-

E.

Segal

Y.

Barash

I.

Simon

N.

Friedman

D.

Koller

Proceedings

of

the

Annual

International

Conference

on

Computational

Molecular

Biology

G.

Myers

et

al.

Eds

-LRB-

ACM

Press

New

York

-RRB-

E.

Segal

R.

Yelensky

D.

Koller

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

P.

T.

Spellman

et

al.

Mol

Biol

Cell

-LRB-

-RRB-

A.

P.

Gasch

et

al.

Mol

Biol

Cell

-LRB-

-RRB-

T.

Lee

et

al.

Science

-LRB-

-RRB-

E.

Segal

H.

Wang

D.

Koller

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

M.

Deng

T.

Chen

F.

Sun

Proceedings

of

the

Annual

International

Conference

on

Computational

Molecular

Biology

M.

Vingron

et

al.

Eds

-LRB-

ACM

Press

New

York

-RRB-

pp.

N.

Friedman

M.

Linial

I.

Nachman

D.

Pe'er

J.

Comp

Biol

-LRB-

-RRB-

D.

Heckerman

D.

Geiger

D.

M.

Chickering

Mach

Learn

-LRB-

-RRB-

P.

Spirtes

C.

Glymour

R.

Scheines

Causation

Prediction

and

Search

-LRB-

Springer-Verlag

Berlin

-RRB-

D.

Pe'er

A.

Regev

G.

Elidan

N.

Friedman

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

D.

Pe'er

A.

Regev

A.

Tanay

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

E.

Segal

et

al.

Nature

Genet

-LRB-

-RRB-

A.

Hartemink

D.

Gifford

T.

S.

Jaakkola

R.

A.

Young

Pac

Symp

Biocomput

-LRB-

-RRB-

S.

Imoto

T.

Goto

S.

Miyano

Pac

Symp

Biocomput

-LRB-

-RRB-

H.

Toh

K.

Horimoto

Bioinformatics

-LRB-

-RRB-

T.

R.

Hughes

et

al.

Cell

-LRB-

-RRB-

S.

Shen-Orr

R.

Milo

S.

Mangan

U.

Alon

Nature

Genet

-LRB-

-RRB-

G.

Cooper

C.

Yoo

Proceedings

of

the

Conference

on

Uncertainty

in

Artificial

Intelligence

K.

Laskey

H.

Prade

Eds

-LRB-

Morgan

Kaufmann

San

Francisco

-RRB-

pp.

I.

Ong

J.

Glasner

D.

Page

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

S.

Y.

Kim

S.

Imoto

S.

Miyano

Brief

Bioinform

-LRB-

-RRB-

B.

E.

Perrin

et

al.

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

A.

J.

Hartemink

D.

K.

Gifford

T.

S.

Jaakkola

R.

A.

Young

Pac

Symp

Biocomput

-LRB-

-RRB-

Y.

Tamada

et

al.

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

T.

Ideker

O.

Ozier

B.

Schwikowski

A.

Siegel

Bioinformatics

-LRB-

suppl

-RRB-

-LRB-

-RRB-

C.

H.

Yeang

T.

Jaakkola

Proceedings

of

the

Annual

International

Conference

on

Computational

Molecular

Biology

M.

Vingron

et

al.

Eds

-LRB-

ACM

Press

New

York

-RRB-

pp.

H.

H.

McAdams

A.

Arkin

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

W.

Vance

A.

Arkin

J.

Ross

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

M.

Ashburner

et

al.

Nature

Genet

-LRB-

-RRB-

I

thank

L.

Garwin

R.

Nelken

D.

Pe'er

A.

Regev

and

I.

Wapinski

for

useful

comments

on

earlier

versions

of

this

manuscript

and

N.

Kaminski

I.

Nachman

D.

Pe'er

A.

Regev

E.

Segal

and

in

particular

D.

Koller

for

many

insightful

discussions

Supported

by

the

Bauer

Center

for

Genomics

Research

Harvard

University

and

a

grant

from

the

Israel

Science

Foundation

O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O