letters

to

nature

smallest

mean

squared

error

This

estimate

is

a

weighted

sum

of

the

mean

of

the

prior

and

the

sensed

feedback

position

xestimated

prior

sensed

cm

xsensed

jsensed

prior

prior

sensed

Sahani

M.

Dayan

P.

Doubly

distributional

population

codes

Simultaneous

representation

of

uncertainty

and

multiplicity

Neural

Comput

-LRB-

-RRB-

Yu

A.

J.

Dayan

P.

Acetylcholine

in

cortical

inference

Neural

Netw

-LRB-

-RRB-

Weiss

Y.

Simoncelli

E.

P.

Adelson

E.

H.

Motion

illusions

as

optimal

percepts

Nature

Neurosci

-LRB-

-RRB-

Soechting

J.

F.

Flanders

M.

Errors

in

pointing

are

due

to

approximations

in

sensorimotor

transformations

J.

Neurophysiol

-LRB-

-RRB-

Krakauer

J.

W.

Pine

Z.

M.

Ghilardi

M.

F.

Ghez

C.

Learning

of

visuomotor

transformations

for

vectorial

planning

of

reaching

trajectories

J.

Neurosci

-LRB-

-RRB-

Lacquaniti

F.

Caminiti

R.

Visuo-motor

transformations

for

arm

reaching

Eur

J.

Neurosci

-LRB-

-RRB-

Van

Beers

R.

J.

Baraduc

P.

Wolpert

D.

M.

Role

of

uncertainty

in

sensorimotor

control

Phil

Trans

R.

Soc

Lond

B

-LRB-

-RRB-

van

Beers

R.

J.

Wolpert

D.

M.

Haggard

P

When

feeling

is

more

important

than

seeing

in

sensorimotor

adaptation

Curr

Biol

-LRB-

-RRB-

Harris

C.

M.

Wolpert

D.

M.

Signal-dependent

noise

determines

motor

planning

Nature

-LRB-

-RRB-

Todorov

E.

Jordan

M.

I.

Optimal

feedback

control

as

a

theory

of

motor

coordination

Nature

Neurosci

-LRB-

-RRB-

Wolpert

D.

M.

Ghahramani

Z.

Jordan

M.

I

An

internal

model

for

sensorimotor

integration

Science

-LRB-

-RRB-

Vetter

P.

Wolpert

D.

M.

Context

estimation

for

sensorimotor

control

J.

Neurophysiol

-LRB-

-RRB-

Scheidt

R.

A.

Dingwell

J.

B.

Mussa-Ivaldi

F.

A.

Learning

to

move

amid

uncertainty

J.

Neurophysiol

-LRB-

-RRB-

Fiorillo

C.

D.

Tobler

P.

N.

Schultz

W.

Discrete

coding

of

reward

probability

and

uncertainty

by

dopamine

neurons

Science

-LRB-

-RRB-

Basso

M.

A.

Wurtz

R.

H.

Modulation

of

neuronal

activity

in

superior

colliculus

by

changes

in

target

probability

J.

Neurosci

-LRB-

-RRB-

Platt

M.

L.

Glimcher

P.

W.

Neural

correlates

of

decision

variables

in

parietal

cortex

Nature

-LRB-

-RRB-

Carpenter

R.

H.

Williams

M.

L.

Neural

computation

of

log

likelihood

in

control

of

saccadic

eye

movements

Nature

-LRB-

-RRB-

Gold

J.

I.

Shadlen

M.

N

The

influence

of

behavioral

context

on

the

representation

of

a

perceptual

decision

in

developing

oculomotor

commands

J.

Neurosci

-LRB-

-RRB-

Goodbody

S.

J.

Wolpert

D.

M.

Temporal

and

amplitude

generalization

in

motor

learning

J.

Neurophysiol

-LRB-

-RRB-

Given

that

we

know

we

can

estimate

the

uncertainty

in

the

feedback

j

sensed

by

prior

linear

regression

from

Fig.

Resulting

mean

squared

error

The

mean

squared

error

-LRB-

MSE

-RRB-

is

determined

by

integrating

the

squared

error

over

all

possible

sensed

feedbacks

and

actual

lateral

shifts

xestimated

xtrue

p

xsensed

jxtrue

p

xtrue

dxsensed

dxtrue

MSE

For

model

x

estimated

x

sensed

and

this

gives

MSE

sensed

Using

the

result

for

x

estimated

from

above

for

model

gives

MSE

sensed

jprior

jsensed

jprior

which

is

always

lower

than

the

MSE

for

model

If

the

variance

of

the

prior

is

equal

to

the

variance

of

the

feedback

the

MSE

for

model

is

half

that

of

model

Inferring

the

used

prior

An

obvious

choice

of

x

estimated

is

the

maximum

of

the

posterior

pffiffiffiffiffiffi

xtrue

p

xtrue

=p

xsensed

p

xtrue

jxsensed

jsensed

The

derivative

of

this

posterior

with

respect

to

x

true

must

vanish

at

x

estimated

This

allows

us

to

estimate

the

prior

used

by

each

subject

Differentiating

and

setting

to

zero

we

get

dp

xtrue

xestimated

xsensed

dxtrue

p

xtrue

xestimated

sensed

We

assume

that

x

sensed

has

a

narrow

peak

around

x

true

and

thus

approximate

it

by

x

true

We

insert

the

j

sensed

obtained

above

affecting

the

scaling

of

the

integral

but

not

its

form

The

average

of

x

sensed

across

many

trials

is

the

imposed

shift

x

true

The

right-hand

side

is

therefore

measured

in

the

experiment

and

the

left-hand

side

approximates

the

derivative

of

log

-LRB-

p

-LRB-

x

true

-RRB-

-RRB-

Since

p

-LRB-

x

true

-RRB-

must

approach

zero

for

both

very

small

and

very

large

x

true

we

subtract

the

mean

of

the

right-hand

side

before

integrating

numerically

to

obtain

log

-LRB-

p

-LRB-

x

true

-RRB-

-RRB-

which

we

can

then

transform

to

estimate

the

prior

p

-LRB-

x

true

-RRB-

Acknowledgements

We

thank

Z.

Ghahramani

for

discussions

and

J.

Ingram

for

technical

support

This

work

was

supported

by

the

Wellcome

Trust

the

McDonnell

Foundation

and

the

Human

Frontiers

Science

Programme

Competing

interests

statement

The

authors

declare

that

they

have

no

competing

financial

interests

Correspondence

and

requests

for

materials

should

be

addressed

to

K.P.K.

-LRB-

email

konrad@koerding.de

-RRB-

Bimodal

distribution

Six

new

subjects

participated

in

a

similar

experiment

in

which

the

lateral

shift

was

bimodally

distributed

as

a

mixture

of

two

gaussians

jprior

x

xsep

jprior

p

xtrue

pffiffiffiffiffiffi

where

x

sep

cm

and

j

prior

cm

Because

we

expected

this

prior

to

be

more

difficult

to

learn

each

subject

performed

trials

split

between

two

consecutive

days

In

addition

to

speed

up

learning

feedback

midway

through

the

movement

was

always

blurred

-LRB-

spheres

distributed

as

a

two-dimensional

gaussian

with

a

standard

deviation

of

cm

-RRB-

and

feedback

at

the

end

of

the

movement

was

provided

on

every

trial

Fitting

the

bayesian

model

-LRB-

using

the

correct

form

of

the

prior

and

true

j

prior

-RRB-

to

minimize

the

MSE

between

actual

and

predicted

lateral

deviations

of

the

last

trials

was

used

to

infer

the

subject

's

internal

estimates

of

both

x

sep

and

j

sensed

Some

aspects

of

the

nonlinear

relationship

between

lateral

shift

and

lateral

deviation

-LRB-

Fig.

-RRB-

can

be

understood

intuitively

When

the

sensed

shift

is

zero

the

actual

shift

is

equally

likely

to

be

to

the

right

or

the

left

and

on

average

there

should

be

no

deviation

from

the

target

If

the

sensed

shift

is

slightly

to

the

right

such

as

at

cm

then

the

actual

shift

is

more

likely

to

come

from

the

right-hand

gaussian

than

the

left

and

subjects

should

point

to

the

right

of

the

target

However

if

the

sensed

shift

is

far

to

the

right

such

as

at

cm

then

because

the

bulk

of

the

prior

lies

to

the

left

subjects

should

point

to

the

left

of

the

target

Received

June

accepted

October

doi

van

Beers

R.

J.

Sittig

A.

C.

Gon

J.

J.

Integration

of

proprioceptive

and

visual

position-information

An

experimentally

supported

model

J.

Neurophysiol

-LRB-

-RRB-

Jacobs

R.

A.

Optimal

integration

of

texture

and

motion

cues

to

depth

Vision

Res

-LRB-

-RRB-

Ernst

M.

O.

Banks

M.

S.

Humans

integrate

visual

and

haptic

information

in

a

statistically

optimal

fashion

Nature

-LRB-

-RRB-

Hillis

J.

M.

Ernst

M.

O.

Banks

M.

S.

Landy

M.

S.

Combining

sensory

information

mandatory

fusion

within

but

not

between

senses

Science

-LRB-

-RRB-

Cox

R.

T.

Probability

frequency

and

reasonable

expectation

Am

J.

Phys

-LRB-

-RRB-

Bernardo

J.

M.

Smith

A.

F.

M.

Bayesian

Theory

-LRB-

Wiley

New

York

-RRB-

Berrou

C.

Glavieux

A.

Thitimajshima

P.

Near

Shannon

limit

error-correcting

coding

and

decoding

turbo-codes

Proc

ICC

Geneva

Switzerland

-LRB-

-RRB-

Simoncelli

E.

P.

Adelson

E.

H.

Noise

removal

via

Bayesian

wavelet

coring

Proc

International

Conference

on

Image

Processing

Lausanne

Switzerland

September

-LRB-

-RRB-

Olshausen

B.

A.

Millman

K.

J.

in

Advances

in

Neural

Information

Processing

Systems

vol

-LRB-

eds

Solla

S.

A.

Leen

T.

K.

Muller

K.

R.

-RRB-

-LRB-

MIT

Press

-RRB-

Rao

R.

P.

N

An

optimal

estimation

approach

to

visual

perception

and

learning

Vision

Res

-LRB-

-RRB-

Hinton

G.

E.

Dayan

P.

Frey

B.

J.

Neal

R.

M

The

wake

sleep

algorithm

for

unsupervised

neural

networks

Science

-LRB-

-RRB-

Functional

genomic

hypothesis

generation

and

experimentation

by

a

robot

scientist

Ross

D.

Kenneth

E.

Ffion

M.

Philip

G.

K.

Christopher

H.

Stephen

H.

Douglas

B.

Stephen

G.

Department

of

Computer

Science

University

of

Wales

Aberystwyth

UK

School

of

Computing

The

Robert

Gordon

University

Aberdeen

UK

Department

of

Computing

Imperial

College

London

UK

Department

of

Chemistry

UMIST

P.O.

Box

Manchester

UK

School

of

Biological

Sciences

University

of

Manchester

Stopford

Building

Manchester

PT

UK

The

question

of

whether

it

is

possible

to

automate

the

scientific

process

is

of

both

great

theoretical

and

increasing

practical

importance

because

in

many

scientific

areas

data

are

being

generated

much

faster

than

they

can

be

effectively

analysed

We

describe

a

physically

implemented

robotic

system

that

applies

techniques

from

artificial

to

carry

out

cycles

of

scientific

experimentation

The

system

automatically

NATURE

VOL

JANUARY

Nature

Publishing

Group

letters

to

nature

originates

hypotheses

to

explain

observations

devises

experiments

to

test

these

hypotheses

physically

runs

the

experiments

using

a

laboratory

robot

interprets

the

results

to

falsify

hypotheses

inconsistent

with

the

data

and

then

repeats

the

cycle

Here

we

apply

the

system

to

the

determination

of

gene

function

using

deletion

mutants

of

yeast

-LRB-

Saccharomyces

cerevisiae

-RRB-

and

auxotrophic

growth

We

built

and

tested

a

detailed

logical

model

-LRB-

involving

genes

proteins

and

metabolites

-RRB-

of

the

aromatic

amino

acid

synthesis

pathway

In

biological

experiments

that

automatically

reconstruct

parts

of

this

model

we

show

that

an

intelligent

experiment

selection

strategy

is

competitive

with

human

performance

and

significantly

outperforms

with

a

cost

decrease

of

and

-LRB-

respectively

-RRB-

both

cheapest

and

random-experiment

selection

The

branch

of

artificial

intelligence

devoted

to

developing

algorithms

for

acquiring

scientific

knowledge

is

known

as

scientific

discovery

The

pioneering

work

in

the

field

was

the

development

of

learning

algorithms

for

analysis

of

mass-spectrometric

In

the

subsequent

years

much

has

been

achieved

and

there

are

now

several

convincing

examples

in

which

computer

programs

have

made

explicit

contributions

to

scientific

However

the

general

impact

of

such

programs

on

science

has

been

limited

This

could

now

change

because

the

expansion

of

automation

in

science

is

making

it

increasingly

possible

to

couple

scientific

discovery

software

to

laboratory

Here

we

describe

a

novel

system

that

extends

the

existing

level

of

integration

of

scientific

discovery

and

robotics

the

Robot

Scientist

-LRB-

Fig.

-RRB-

A

widely

accepted

view

of

science

is

that

it

follows

a

hypotheticodeductive

Scientific

expertise

and

imagination

are

first

used

to

form

possible

hypotheses

and

then

the

deductive

consequences

of

these

hypotheses

are

tested

by

experiment

The

Robot

Scientist

follows

this

paradigm

we

employ

the

logical

inference

mechanism

of

to

form

new

hypotheses

and

that

of

deduction

to

test

which

hypotheses

are

consistent

-LRB-

see

Methods

and

Supplementary

Information

-RRB-

The

system

has

been

physically

implemented

and

conducts

biological

assays

with

minimal

human

intervention

after

the

robot

is

set

up

The

hardware

platform

consists

of

a

liquid-handling

robot

with

its

control

PC

a

plate

reader

with

its

control

PC

and

a

master

PC

to

control

the

system

and

do

the

scientific

reasoning

The

software

platform

consists

of

background

knowledge

about

the

biological

problem

a

logical

inference

engine

hypothesis

generation

code

-LRB-

abduction

-RRB-

experiment

selection

code

-LRB-

deduction

-RRB-

and

the

Laboratory

Information

Management

System

-LRB-

LIMS

-RRB-

code

that

integrates

the

whole

system

The

robot

conducts

assays

by

pipetting

and

mixing

liquids

on

microtitre

plates

Given

a

computed

definition

of

one

or

more

experiments

we

have

developed

code

that

designs

a

layout

of

reagents

on

the

liquid-handling

platform

that

will

allow

these

experiments

with

controls

to

be

performed

efficiently

In

addition

the

liquid-handling

robot

is

automatically

programmed

to

plate

out

the

yeast

and

media

into

the

correct

wells

The

system

measures

the

concentration

of

yeast

in

the

wells

of

the

microtitre

trays

using

the

adjacent

plate

reader

and

returns

the

results

to

the

LIMS

-LRB-

although

microtitre

trays

are

still

moved

in

and

out

of

incubators

manually

-RRB-

The

key

point

is

that

there

was

no

human

intellectual

input

in

the

design

of

experiments

or

the

interpretation

of

data

To

test

the

Robot

Scientist

we

chose

the

field

of

functional

genomics

It

is

one

of

the

most

important

in

contemporary

science

and

is

an

area

in

which

laboratory

automation

is

already

mature

The

current

state

of

the

art

in

functional

genomics

is

to

use

highly

automated

robotics

to

generate

data

and

then

to

use

data-mining

systems

to

extract

knowledge

from

that

data

Within

functional

genomics

we

selected

yeast

-LRB-

S.

cerevisiae

-RRB-

to

test

the

methodology

Despite

being

one

of

the

best

understood

of

organisms

the

functions

of

about

of

the

genes

in

yeast

are

still

The

aim

was

to

develop

a

system

that

could

automatically

determine

the

function

of

genes

from

the

performance

of

knockout

mutants

-LRB-

strains

in

which

one

gene

has

been

removed

-RRB-

We

focused

on

the

aromatic

amino

acid

synthesis

-LRB-

AAA

-RRB-

pathway

-LRB-

Fig.

-RRB-

and

used

auxotrophic

growth

experiments

to

assess

the

behaviour

-LRB-

phenotype

-RRB-

of

the

mutants

The

AAA

pathway

is

relatively

well

understood

and

of

sufficient

complexity

to

make

reasoning

about

it

non-trivial

and

its

intermediary

metabolites

are

commercially

available

Auxotrophic

growth

experiments

consist

of

growing

auxotrophic

mutants

on

chemically

defined

media

-LRB-

a

defined

base

plus

one

or

more

intermediate

or

terminal

metabolites

in

the

pathway

-RRB-

and

observing

whether

growth

is

recovered

or

not

-LRB-

see

Supplementary

Information

for

details

-RRB-

A

knockout

mutant

is

auxotrophic

if

it

can

not

grow

on

a

defined

medium

on

which

the

wild

type

can

grow

Auxotrophic

experiments

are

a

classic

technique

for

inferring

metabolic

We

needed

a

way

of

representing

prior

biological

knowledge

in

the

computer

so

we

developed

a

logical

formalism

to

model

cellular

metabolism

that

captures

the

key

relationship

between

proteincoding

sequences

-LRB-

open

reading

frames

ORFs

-RRB-

enzymes

and

metabolites

in

a

All

objects

-LRB-

ORFs

proteins

and

metabolites

-RRB-

and

relationships

-LRB-

coding

reactions

transport

and

feedback

-RRB-

are

described

as

logical

formulae

The

structure

of

the

metabolic

pathway

is

that

of

a

directed

graph

with

metabolites

as

nodes

and

enzymes

as

arcs

An

arc

corresponds

to

a

reaction

The

compounds

at

each

node

are

the

set

of

all

metabolites

that

can

be

synthesized

by

the

reactions

leading

to

it

Reactions

are

modelled

as

unidirectional

transformations

Using

this

formalism

we

have

implemented

a

model

of

the

AAA

pathway

with

the

logic

programming

language

Prolog

-LRB-

the

complete

model

is

provided

in

Supplementary

Information

-RRB-

Prolog

makes

it

possible

both

to

inspect

the

biological

knowledge

in

the

model

directly

and

to

compute

the

predictions

of

the

model

automatically

The

model

infers

-LRB-

deduces

-RRB-

that

a

knockout

mutant

will

grow

if

and

only

if

a

path

can

be

found

from

the

input

metabolites

to

the

three

aromatic

amino

acids

This

allows

the

model

to

compute

the

phenotype

of

a

particular

knockout

or

to

be

used

to

infer

missing

reactions

that

could

explain

an

observed

phenotype

-LRB-

abduction

-RRB-

We

consider

that

most

hypothesis

generation

in

modern

biology

is

abductive

What

is

inferred

are

not

general

hypotheses

which

would

be

inductive

but

specific

facts

about

biological

entities

The

original

bioinformatic

information

for

the

AAA

model

was

taken

mainly

from

the

catalogue

of

metabolism

The

model

was

then

tested

with

all

possible

auxotrophic

experiments

involving

a

single

replacement

metabolite

and

was

altered

manually

to

fit

the

empirical

results

To

ensure

that

the

model

was

not

over-fitted

we

carried

out

all

possible

auxotrophic

experiments

with

pairs

of

metabolites

The

model

correctly

predicted

at

least

of

the

experiments

-LRB-

Supplementary

Information

-RRB-

To

the

best

of

our

knowledge

no

bioinformatic

model

has

been

as

thoroughly

tested

with

knockout

mutants

Machine

learning

is

the

branch

of

artificial

intelligence

that

seeks

to

develop

computer

systems

that

improve

their

performance

automatically

with

It

has

much

in

common

with

statistics

but

differs

in

having

a

greater

emphasis

on

algorithms

data

representation

and

making

acquired

knowledge

explicit

The

NATURE

VOL

JANUARY

Figure

The

Robot

Scientist

hypothesis-generation

and

experimentation

loop

Nature

Publishing

Group

letters

to

nature

branch

of

machine

learning

that

deals

with

algorithms

that

can

choose

experiments

is

known

as

active

learning

If

we

assume

that

each

hypothesis

has

a

prior

probability

of

being

correct

and

that

each

experiment

has

an

associated

cost

then

scientific

experiment

selection

can

be

formalized

as

the

task

of

selecting

the

optimal

series

of

experiments

-LRB-

in

terms

of

expected

cost

-RRB-

to

eliminate

all

except

the

one

correct

hypothesis

This

problem

is

in

general

computationally

intractable

-LRB-

NP-hard

-RRB-

However

it

can

be

shown

that

the

problem

is

structurally

the

same

as

finding

the

smallest

decision

tree

experiments

are

nodes

and

hypotheses

are

This

is

significant

because

a

bayesian

analysis

of

decision-tree

learning

has

shown

that

near-optimal

solutions

can

be

found

in

polynomial

This

analysis

leads

to

the

following

approximate

recurrence

formula

for

the

expected

Let

EC

-LRB-

H

T

-RRB-

denote

the

minimum

expected

cost

of

experimentation

given

the

set

of

candidate

hypotheses

H

and

the

set

of

candidate

trials

T

EC

B

T

EC

-LCB-

h

-RCB-

T

EC

H

T

mint

-LSB-

T

C

t

p

t

meant

-LSB-

Ct

J

H

t

p

t

meant

-LSB-

C

t

J

H

t

J

H

-LSB-

H

p

h

p

h

c

where

C

t

is

the

monetary

price

of

the

trial

t

p

-LRB-

t

-RRB-

is

the

probability

that

the

outcome

of

the

trial

t

is

positive

and

b.

c

is

the

floor

function

p

-LRB-

t

-RRB-

can

be

computed

as

the

sum

of

the

probabilities

of

the

hypotheses

-LRB-

h

-RRB-

that

are

consistent

with

a

positive

outcome

of

t

In

current

robot-scientist

experiments

it

is

assumed

that

the

system

knows

the

biochemical

equations

of

the

AAA

pathway

in

the

wild

type

What

the

system

does

not

know

is

how

this

biochemistry

is

related

to

the

genetics

The

hypotheses

are

therefore

bindings

between

the

ORFs

and

enzymic

reactions

that

is

which

of

the

possible

ORFs

in

the

pathway

has

been

deleted

-LRB-

although

only

ORFs

were

ever

actually

used

for

technical

reasons

the

system

was

not

aware

of

this

-RRB-

For

example

a

correct

hypothesis

would

be

that

ORF

codes

for

chorismate

mutase

-LRB-

which

catalyses

the

reaction

chorismate

prephenate

Fig.

-RRB-

These

hypotheses

are

automatically

generated

by

abducing

the

different

possibilities

from

the

Hypotheses

are

evaluated

by

comparing

their

predicted

consequences

with

the

actual

experimental

results

These

logical

inferences

are

made

with

our

machine

learning

system

where

ASE

stands

for

active

selection

of

experiments

-LRB-

ASE-Progol

is

freely

available

to

academics

from

kftp

-RRB-

We

compared

three

experimental

selection

strategies

first

ASE

-LRB-

our

approximation

to

choosing

the

experiment

that

minimizes

the

expected

cost

see

above

-RRB-

second

Naive

-LRB-

which

chooses

the

cheapest

experiment

that

has

not

yet

been

done

-RRB-

and

last

Random

-LRB-

which

chooses

a

random

experiment

that

has

not

yet

been

done

-RRB-

The

performance

of

a

strategy

is

defined

as

the

average

accuracy

of

all

hypotheses

not

already

eliminated

by

experiments

using

that

strategy

The

accuracy

of

a

single

hypothesis

is

the

number

of

correct

predictions

that

this

hypothesis

makes

for

all

possible

singlemetabolite

and

double-metabolite

addition

experiments

assuming

that

the

model

is

accurate

We

evaluated

accuracy

versus

the

monetary

price

of

the

experiments

-LRB-

pounds

sterling

-RRB-

and

time

-LRB-

iterations

that

is

the

number

of

sequential

experiments

-RRB-

Using

the

Robot

Scientist

we

compared

the

experimental

strategies

ASE

Naive

and

Random

over

five

experimental

iterations

-LRB-

Fig.

b

-RRB-

At

the

end

of

the

fifth

iteration

ASE

had

an

estimated

accuracy

of

in

comparison

with

for

Naive

and

for

Random

At

this

point

in

the

iterative

cycle

of

experimentation

the

ASE

strategy

was

significantly

more

accurate

than

either

Naive

-LRB-

P

-RRB-

or

Random

-LRB-

P

-RRB-

using

a

paired

t-test

Given

a

maximum

spend

of

-LRB-

the

maximum

that

Naive

spends

note

that

these

prices

are

scaled

-RRB-

ASE

has

an

estimated

accuracy

of

in

comparison

with

for

Naive

and

for

Random

For

this

price

ASE

is

significantly

more

accurate

than

either

Naive

-LRB-

P

-RRB-

or

Random

-LRB-

P

-RRB-

Although

the

AAA

pathway

is

sufficiently

complex

to

provide

an

adequate

test

for

the

robot

scientist

system

it

is

not

so

complex

that

the

Naive

and

Random

strategies

will

not

eventually

achieve

a

high

accuracy

Where

ASE

shows

its

superiority

over

the

other

two

experimental

strategies

is

in

Figure

A

schematic

representation

of

the

logical

model

of

the

aromatic

amino

acid

pathway

in

yeast

The

metabolites

are

the

nodes

and

the

enzymes

the

arcs

in

the

directed

graph

Metabolites

that

could

not

be

used

in

auxotrophic

experiments

are

in

italics

and

NATURE

VOL

JANUARY

the

end-product

aromatic

amino

acids

are

shown

in

green

The

ORFs

encoding

the

enzyme

proteins

are

blue

if

they

are

amenable

to

auxotrophic

experimentation

otherwise

they

are

red

The

logical

model

also

represents

the

import

of

metabolites

into

the

cell

Nature

Publishing

Group

letters

to

nature

cost-effectiveness

In

science

as

in

industry

time

is

money

-LRB-

although

the

conversion

rate

may

be

unclear

-RRB-

The

experimental

cost

we

wish

to

optimize

will

therefore

be

a

function

of

price

and

time

Because

ASE

dominates

for

both

ASE

is

superior

to

Naive

and

Random

for

any

positive

function

of

price

and

time

For

example

to

achieve

an

accuracy

of

about

ASE

requires

fewer

trial

iterations

and

a

hundredth

of

the

price

of

Random

and

almost

half

the

number

of

iterations

and

a

third

of

the

price

of

Naive

We

evaluated

the

reliability

of

these

empirical

results

by

running

several

computer

simulations

of

the

Robot

Scientist

In

these

we

used

the

AAA

Prolog

model

and

the

ASE-Progol

inference

system

and

we

took

the

results

of

manual

experiments

When

necessary

we

also

added

random

noise

by

flipping

the

observation

of

growth

to

no-growth

and

the

reverse

We

compared

differing

types

of

experiment

-LRB-

single

metabolites

or

double

metabolites

-RRB-

numbers

of

iterations

noise

and

noise

reduction

strategies

In

Fig.

d

we

show

the

results

for

and

experimental

noise

plotting

accuracy

against

time

and

accuracy

against

price

In

these

typical

experiments

the

ASE

strategy

almost

completely

dominates

those

of

Naive

and

Random

These

simulation

experiments

are

consistent

with

the

results

in

vivo

because

we

estimate

that

the

physical

system

has

about

noise

in

the

observations

of

whether

growth

occurred

or

not

The

high

level

of

noise

is

due

to

the

robot

's

being

open

to

the

air

-LRB-

and

therefore

to

microbial

contamination

-RRB-

We

were

also

interested

to

compare

the

performance

of

the

Robot

Scientist

system

with

that

of

humans

To

do

this

we

adapted

the

simulator

to

allow

humans

to

choose

and

interpret

the

result

of

cycles

of

experimentation

In

initial

trials

using

nine

graduate

computer

scientists

and

biologists

we

found

that

there

was

no

significant

difference

between

the

robot

and

the

best

human

performance

in

terms

of

the

number

of

iterations

required

to

achieve

a

given

level

of

accuracy

The

inference

of

gene

function

in

the

AAA

pathway

was

selected

to

be

typical

of

scientific

inference

tasks

in

molecular

genetics

We

argue

that

most

molecular

genetics

differs

from

this

model

only

in

the

much

greater

sophistication

of

the

abductive

process

used

to

select

a

possible

hypothesis

and

by

the

far

greater

elegance

of

the

experiments

used

to

discriminate

among

hypotheses

Although

the

AAA

problem

is

relatively

simple

it

could

still

be

applied

directly

to

real-world

problems

For

instance

if

a

series

of

AAA

metabolism

mutants

from

a

related

-LRB-

but

genetically

uncharacterized

-RRB-

fungal

species

had

been

isolated

the

S.

cerevisiae

AAA

pathway

could

be

used

to

guide

auxotrophic

growth

experiments

in

exactly

the

same

way

as

described

here

Although

pioneering

work

by

others

-LRB-

see

ref

for

example

-RRB-

has

developed

automated

systems

to

induce

genetic

networks

from

mutant

data

we

believe

that

the

Robot

Scientist

is

the

first

example

of

a

closed-loop

system

that

actually

designs

and

executes

experiments

to

test

inferred

hypotheses

Nevertheless

the

Robot

Scientist

has

currently

only

been

demonstrated

to

rediscover

the

role

of

genes

of

known

function

this

initial

step

was

essential

to

ensure

that

the

system

was

working

We

now

plan

to

extend

the

system

to

be

able

to

uncover

the

function

of

genes

whose

role

is

currently

unknown

To

enable

this

the

background

cellular

metabolism

model

will

need

to

be

extended

This

will

involve

the

translation

of

bioinformatic

databases

such

as

KEGG

into

our

logical

formalism

the

resultant

models

are

likely

to

be

less

reliable

than

that

of

our

carefully

checked

AAA

model

The

abductive

hypothesis

generation

method

will

also

need

to

be

extended

to

allow

the

inference

of

missing

arcs

not

just

their

binding

Initial

work

has

been

done

on

The

general

robot

scientist

idea

could

be

applied

to

many

scientific

problems

and

we

are

actively

investigating

drug

design

Figure

Actual

and

simulated

performance

of

the

Robot

Scientist

a

b

Actual

performance

a

Plot

of

average

classification

accuracy

against

experimental

iteration

-LRB-

from

four

experimental

repeats

-RRB-

b

Plot

of

average

classification

accuracy

against

average

experimental

scaled

price

-LRB-

from

four

experimental

repeats

-RRB-

c

d

Simulated

performance

c

Plot

of

average

classification

accuracy

against

time

for

noise

-LRB-

squares

-RRB-

and

noise

-LRB-

diamonds

-RRB-

-LRB-

based

on

simulation

runs

-RRB-

d

Plot

of

average

classification

accuracy

against

average

scaled

price

for

noise

-LRB-

squares

-RRB-

and

noise

-LRB-

diamonds

-RRB-

-LRB-

based

on

simulation

runs

-RRB-

Strategies

red

curves

ASE

blue

curves

Random

green

curves

Naive

NATURE

VOL

JANUARY

Nature

Publishing

Group

letters

to

nature

and

quantum

control

To

apply

the

Robot

Scientist

to

drug

design

the

robot

would

need

to

be

able

to

perform

a

biological

assay

to

perform

the

given

optimization

and

either

synthesize

compounds

of

required

structure

using

lab-on-a-chip

technology

or

choose

them

from

a

large

library

The

hypotheses

would

be

different

qualitative

structure

activity

relationships

The

quantum

control

of

chemical

synthesis

with

femtosecond

is

especially

challenging

because

of

the

ability

and

desirability

of

doing

experiments

very

quickly

-LRB-

about

per

second

at

present

-RRB-

At

such

speeds

human

intervention

is

impossible

and

automatic

systems

become

essential

The

Robot

Scientist

extends

the

state

of

the

art

in

integrating

scientific

discovery

software

with

laboratory

robotics

Moreover

the

application

of

the

Robot

Scientist

to

functional

genomics

provides

further

evidence

that

some

aspects

of

scientific

reasoning

can

be

formalized

and

efficiently

automated

In

science

experiment

selection

is

generally

done

informally

and

without

explicit

regard

to

the

cost

of

eliminating

hypotheses

when

formal

experimental

is

employed

this

is

generally

done

to

ensure

that

sufficient

repeats

and

controls

are

in

place

to

answer

the

desired

question

Our

results

show

that

different

experiment

selection

strategies

can

have

significantly

different

results

in

terms

of

cost

even

for

the

solution

of

a

simple

problem

This

suggests

that

there

remains

scope

to

improve

the

general

cost-effectiveness

of

science

by

developing

better

tools

to

help

choose

efficient

experiments

Automation

was

the

driving

force

of

much

of

nineteenth-century

and

twentieth-century

change

and

this

is

likely

to

continue

It

is

also

becoming

increasingly

important

in

scientific

research

for

example

the

sequencing

of

the

human

genome

was

made

possible

by

factory

production

techniques

and

modern

drug

discovery

relies

on

high-throughput

screening

robots

We

consider

this

trend

to

increased

automation

of

science

to

be

both

inevitable

and

desirable

It

is

inevitable

because

it

will

be

required

to

deal

with

the

challenges

of

science

in

the

twenty-first

century

It

is

also

desirable

because

it

frees

scientists

to

make

the

high-level

creative

leaps

at

which

they

excel

A

because

it

rewards

learners

that

discriminate

between

competing

hypotheses

This

approach

is

a

compromise

between

selecting

the

hypothesis

with

the

highest

probability

and

weighting

all

predictions

by

the

probability

of

the

hypotheses

that

generated

them

In

active

learning

the

performance

curves

that

have

been

generally

used

plot

predictive

accuracy

against

the

number

of

training

examples

Often

two

curves

are

plotted

on

the

same

graph

one

for

active

learning

and

one

for

random

sampling

The

accuracy

of

a

single

hypothesis

is

the

number

of

correct

predictions

that

this

hypothesis

makes

about

all

possible

single-metabolite

and

double-metabolite

experiments

based

on

using

the

model

as

the

oracle

Such

performance

plots

allow

the

difference

in

the

number

of

experiments

-LRB-

examples

per

unit

time

-RRB-

required

to

reach

a

particular

level

of

performance

to

be

compared

However

one

drawback

of

such

plots

is

that

they

ignore

any

variation

in

the

price

of

obtaining

individual

examples

When

such

variation

does

exist

and

the

aim

is

to

compare

the

price

of

attaining

particular

levels

of

performance

these

plots

are

potentially

misleading

To

overcome

this

drawback

we

also

plot

the

cumulative

price

of

the

experiments

against

For

this

we

use

the

normalized

price

of

the

metabolite

At

the

start

of

the

experiments

when

there

are

eight

possible

hypotheses

the

average

accuracy

is

Structure

of

the

Robot

Scientist

The

Robot

Scientist

automates

the

task

of

liquid

handling

and

can

conduct

assays

by

pipetting

and

mixing

liquids

on

microtitre

plates

The

robot

is

controlled

using

Tcl

-LRB-

Tool

Command

Language

-RRB-

and

we

have

written

a

compiler

that

translates

Prolog

commands

into

Tcl

robot

operations

Given

a

Prolog

definition

of

one

or

more

experiments

we

have

developed

code

that

designs

a

layout

of

the

robot

that

will

allow

these

experiments

with

controls

to

be

performed

efficiently

In

addition

the

robot

has

to

be

automatically

programmed

to

plate

out

the

yeast

and

media

into

the

correct

wells

The

microtitre

plates

were

measured

with

the

adjacent

plate

reader

and

the

results

were

returned

to

the

LIMS

Transfer

of

plates

from

the

robot

to

the

incubator

and

from

the

incubator

to

the

plate

reader

was

done

manually

Received

July

accepted

November

doi

Popper

K

The

Logic

of

Scientific

Discovery

-LRB-

Hutchinson

London

-RRB-

Sloman

A

The

Computer

Revolution

in

Philosophy

-LRB-

Harvester

Hassocks

Sussex

-RRB-

available

online

from

khttp

Buchanan

B.

G.

Sutherland

G.

L.

Feigenbaum

E.

A.

in

Machine

Intelligence

Vol

-LRB-

eds

Meltzer

B.

Michie

D.

-RRB-

-LRB-

Edinburgh

Univ.

Press

-RRB-

Langley

P.

Simon

H.

A.

Bradshaw

G.

L.

Zytkow

J.

M.

Scientific

Discovery

Computational

Explorations

of

the

Creative

Process

-LRB-

MIT

Press

Cambridge

Massachusetts

-RRB-

Zytkow

J.

M.

Zhu

J.

Hussam

A.

Automated

discovery

in

a

chemistry

laboratory

in

Proceedings

of

the

National

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

-LRB-

eds

Dietterich

T.

Swartout

W.

-RRB-

-LRB-

MIT

Cambridge

Massachusetts

-RRB-

King

R.

D.

Muggleton

S.

H.

Srinivasan

A.

Sternberg

M.

J.

E.

Structure

activity

relationships

derived

by

machine

learning

The

use

of

atoms

and

their

bond

connectivities

to

predict

mutagenicity

by

inductive

logic

programming

Proc

Natl

Acad

Sci

USA

-LRB-

-RRB-

Valdes-Perez

R.

E.

Discovery

tools

for

science

applications

Commun

ACM

-LRB-

-RRB-

Langley

P

The

computational

support

of

scientific

discovery

Int

J.

Hum

Comput

Stud

-LRB-

-RRB-

Beadle

G.

W.

Tatum

E.

I.

Genetic

control

of

biochemical

reactions

in

Neurospora

Proc

Natl

Acad

Sci

USA

-LRB-

-RRB-

Pierce

C.

S.

Collected

Papers

of

Charles

Sanders

Pierce

-LRB-

Harvard

Univ.

Press

-RRB-

Mewes

H.

W.

et

al.

MIPS

a

database

for

genomes

and

protein

sequences

Nucleic

Acids

Res

-LRB-

-RRB-

khttp

mips.gsf.de

proj/yeast/CYGD

db/index

htmll

Reiser

P.

G.

K.

et

al.

Developing

a

logical

model

of

yeast

metabolism

Electron

Trans

Artif

Intell

-LRB-

-RRB-

Kanehisa

M.

Goto

S.

KEGG

Kyoto

Encyclopedia

of

Genes

and

Genomes

Nucleic

Acids

Res

-LRB-

-RRB-

khttp

Mitchell

T.

M.

Machine

Learning

-LRB-

McGraw-Hill

New

York

-RRB-

Langley

P.

Elements

of

Machine

Learning

-LRB-

Morgan

Kaufmann

San

Mateo

California

-RRB-

Cohen

D.

A.

Ghabhramani

Z.

Jordan

M.

I.

Active

learning

with

statistical

models

J.

Artif

Intell

Res

-LRB-

-RRB-

Lin

F.-R

Shaw

M.

J.

Active

training

of

backpropagation

neural

networks

using

the

learning

by

experimentation

methodology

Ann

Oper

Res

-LRB-

-RRB-

Fedorov

V.

V.

Theory

of

Optimal

Experiments

-LRB-

Academic

London

-RRB-

Muggleton

S.

Page

D.

in

Machine

Intelligence

Vol

-LRB-

eds

Furukawa

K.

Michie

D.

Muggleton

S.

-RRB-

-LRB-

Oxford

Univ.

Press

-RRB-

Bryant

C.

H.

Muggleton

S.

H.

Oliver

S.

G.

Kell

D.

B.

Reiser

P.

G.

K.

King

R.

D.

Combining

inductive

logic

programming

active

learning

and

robotics

to

discover

the

function

of

genes

Electron

Trans

Artif

Intell

-LRB-

-RRB-

Flach

P.

Kaka

A.

Abduction

and

Induction

-LRB-

Kluwer

London

-RRB-

Zupan

B.

et

al.

in

Proceedings

of

the

Eighth

European

Conference

on

Artificial

Intelligence

in

Medicine

-LRB-

eds

Qualini

S.

Barahona

P.

Andreassen

S.

-RRB-

-LRB-

Springer

Berlin

-RRB-

Muggleton

S.

Inverse

entailment

and

Progol

New

Generation

Comput

J.

-LRB-

-RRB-

Rabitz

H.

de

Vivie-Riedle

R.

Motzkus

M.

Kompa

K.

Whither

the

future

of

controlling

quantum

phenomena

Science

-LRB-

-RRB-

Cochran

W.

G.

Cox

G.

M.

Experimental

Designs

-LRB-

Wiley

New

York

-RRB-

Brachmann

C.

B.

et

al.

Designer

deletion

strains

derived

from

Saccharomyces

cerevisiae

a

useful

set

of

strains

and

plasmids

for

PCR-mediated

gene

disruption

and

other

applications

Yeast

-LRB-

-RRB-

Giaever

G.

et

al.

Functional

profiling

of

the

Saccharomyces

cerevisiae

genome

Nature

-LRB-

-RRB-

Methods

Growth

experiments

and

the

amino

acid

pathway

The

mutants

-LRB-

of

Saccharomyces

cerevisiae

strain

-LRB-

-RRB-

MATa

ref

-RRB-

had

the

complete

reading

frame

of

each

proteinencoding

gene

deleted

by

replacement

with

a

selectable

marker

gene

that

had

no

phenotype

in

the

absence

of

the

selective

They

were

therefore

non-revertible

mutants

Limitations

in

the

availability

of

mutant

yeast

strains

restricted

the

number

of

genes

used

in

the

in

vivo

investigations

to

and

the

auxotrophic

experimental

requirement

for

a

difference

in

growth

phenotype

reduced

this

number

to

the

following

-LRB-

the

other

mutants

always

either

grew

or

failed

to

grow

-RRB-

and

The

number

of

possible

metabolites

was

limited

to

nine

by

availability

and

cost

anthranilate

-LRB-

-RRB-

indole

-LRB-

-RRB-

p-hydroxyphenolpyruvic

acid

-LRB-

-RRB-

L

phenylalanine

-LRB-

-RRB-

phenylpyruvate

-LRB-

-RRB-

phosphoenolpyruvate

-LRB-

-RRB-

shikimic

acid

-LRB-

-RRB-

L-tyrosine

-LRB-

-RRB-

and

L

tryptophan

-LRB-

-RRB-

The

numbers

in

parentheses

are

the

normalized

true

experimental

costs

of

using

each

metabolite

in

a

growth

medium

note

the

approximate

three

orders

of

magnitude

range

Computational

model

of

the

aromatic

amino

acid

pathway

The

Prolog

model

of

the

pathway

was

refined

in

three

stages

First

the

original

model

was

translated

from

and

carefully

checked

with

the

literature

We

checked

that

all

the

KEGG

reactions

were

documented

in

S.

cerevisiae

-LRB-

consistency

-RRB-

and

that

there

were

no

other

related

reactions

described

in

the

literature

-LRB-

completeness

-RRB-

Second

the

predictions

of

the

model

were

compared

with

the

results

of

the

single-metabolite

experiments

-LRB-

see

above

-RRB-

Whether

growth

or

lack

of

growth

was

observed

was

at

this

point

decided

visually

Because

certain

metabolites

did

not

seem

to

affect

growth

in

the

way

predicted

by

the

literature

we

refined

the

model

to

make

these

metabolites

unable

to

be

imported

into

the

cells

efficiently

It

was

also

necessary

to

add

inhibition

effects

-LRB-

see

Supplementary

Information

for

details

-RRB-

Last

the

results

of

the

double-metabolite

experiments

were

tested

against

the

model

and

the

automatic

growth-calling

software

was

optimized

by

learning

a

decision

tree

to

fit

the

experimental

results

to

the

model

-LRB-

see

Supplementary

section

-RRB-

The

model

developed

on

the

single

metabolites

was

consistent

with

all

except

less

than

of

the

double-metabolite

experiments

The

model

was

not

further

changed

to

include

these

experimental

discrepancies

Performance

measures

The

average

performance

of

the

hypotheses

is

an

appropriate

performance

measure

NATURE

VOL

JANUARY

Supplementary

Information

accompanies

the

paper

on

Nature

Publishing

Group

letters

to

nature

Acknowledgements

We

thank

D.

Page

U.

Sarkans

A.

Tamaddoni

M.

Sternberg

A.

Sloman

and

D.

Michie

for

their

help

and

advice

and

D.

Struttman

for

technical

assistance

The

work

was

funded

by

the

BBSRC

the

EPSRC

the

Wellcome

Trust

and

PharmDM

Competing

interests

statement

The

authors

declare

competing

financial

interests

details

accompany

the

paper

on

Correspondence

and

requests

for

materials

should

be

addressed

to

S.G.O.

-LRB-

steve.oliver@man.ac.uk

-RRB-

Immune

recognition

of

a

human

renal

cancer

antigen

through

post-translational

protein

splicing

Ken-ichi

Jonathan

W.

James

C.

Surgery

Branch

National

Cancer

Institute

National

Institutes

of

Health

Rockville

Pike

Room

Bethesda

Maryland

USA

Laboratory

of

Viral

Diseases

National

Institute

of

Allergy

and

Infectious

Diseases

National

Institutes

of

Health

Bethesda

Maryland

USA

Cytotoxic

T

lymphocytes

-LRB-

CTLs

-RRB-

detect

and

destroy

cells

displaying

class

I

molecules

of

the

major

histocompatibility

complex

-LRB-

MHC

-RRB-

that

present

oligopeptides

derived

from

aberrant

self

or

foreign

proteins

Most

class

I

peptide

ligands

are

created

from

proteins

that

are

degraded

by

proteasomes

and

transported

by

the

transporter

associated

with

antigen

processing

from

the

cytosol

into

the

endoplasmic

reticulum

where

peptides

bind

MHC

class

I

molecules

and

are

conveyed

to

the

cell

CTLs

cloned

from

human

CTLs

infiltrating

a

renal

cell

carcinoma

kill

cancer

cells

overexpressing

fibroblast

growth

-LRB-

-RRB-

Here

we

show

that

cells

recognize

human

leukocyte

MHC

class

I

molecules

presenting

a

nine-residue

peptide

generated

by

protein

splicing

This

process

previously

described

strictly

in

and

unicellular

entails

post-translational

excision

of

a

polypeptide

segment

followed

by

ligation

of

the

newly

liberated

carboxyterminal

and

amino-terminal

residues

The

occurrence

of

protein

splicing

in

vertebrates

has

important

implications

for

the

complexity

of

the

vertebrate

proteome

and

for

the

immune

recognition

of

self

and

foreign

peptides

To

identify

the

epitope

recognized

by

we

transfected

human

leukocyte

antigen

-LRB-

HLA

-RRB-

cells

-LRB-

-RRB-

with

expression

plasmids

encoding

truncated

forms

of

the

gene

and

measured

activation

by

its

release

of

interferon-g

-LRB-

IFN-g

-RRB-

-LRB-

Fig.

-RRB-

Antigenicity

was

lost

by

truncation

either

between

predicted

amino

acid

residues

and

-LRB-

plasmids

-RRB-

or

residues

and

-LRB-

plasmids

-RRB-

of

The

shortest

stimulatory

fragment

identified

-LRB-

-RRB-

encoded

amino

acids

-LRB-

residues

-RRB-

We

next

examined

the

antigenicity

of

synthetic

peptides

corresponding

to

all

of

the

and

peptides

contained

within

the

defined

amino-acid

fragment

but

were

recognized

by

when

incubated

with

APCs

expressing

-LRB-

data

not

shown

-RRB-

We

similarly

tested

peptides

encoded

by

the

and

reading

frames

of

the

fragment

but

did

not

find

the

epitope

-LRB-

data

not

shown

-RRB-

Post-translational

structural

modification

of

a

minimal

determinant

within

the

amino-acid

fragment

remained

a

possible

explanation

for

our

failure

to

identify

a

Figure

Genetic

truncation

analysis

of

the

determinant

a

A

series

of

truncation

mutants

-LRB-

-RRB-

were

prepared

by

PCR

from

construct

An

initiation

codon

and

the

Kozak

consensus

-LRB-

CACCATG

-RRB-

were

introduced

into

the

PCR

primers

for

constructs

Each

PCR

product

was

cloned

into

the

eukaryotic

expression

vector

and

transfected

into

cells

recognition

by

was

assessed

by

IFN

g

secretion

Nucleic

acid

positions

are

indicated

above

the

start

and

finish

of

each

construct

and

the

corresponding

amino

acid

positions

from

below

Constructs

shown

as

open

boxes

were

recognized

those

shown

as

filled

boxes

were

not

b

Analysis

of

internal

deletion

mutants

For

simplicity

predicted

amino

acid

sequences

instead

of

DNA

sequences

are

shown

The

antigenicity

of

each

construct

was

assessed

as

described

in

a.

All

constructs

were

designed

to

start

with

the

Kozak

consensus

sequence

and

end

with

the

termination

codon

TAA

Numbers

in

parentheses

indicate

the

position

of

the

first

and

the

last

amino

acids

Numbers

after

D

indicate

internally

deleted

amino

acid

residues

The

degree

of

CTL

recognition

is

categorized

as

follows

depending

on

IFN

g

secretion

-LRB-

-RRB-

more

than

pg

-LRB-

-RRB-

pg

-LRB-

-RRB-

pg

-LRB-

-RRB-

less

than

pg

NATURE

VOL

JANUARY

Nature

Publishing

Group

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

systems for experimental biological design