Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Parameter

Estimation

in

Biochemical

Pathways

A

Comparison

of

Global

Optimization

Methods

Carmen

G.

Moles

Pedro

Mendes

and

Julio

R.

Banga

Genome

Res

originally

published

online

Oct

Access

the

most

recent

version

at

doi

gr

References

This

article

cites

articles

of

which

can

be

accessed

free

at

Article

cited

in

Email

alerting

service

Receive

free

email

alerts

when

new

articles

cite

this

article

sign

up

in

the

box

at

the

top

right

corner

of

the

article

or

click

here

Notes

To

subscribe

to

Genome

Research

go

to

Cold

Spring

Harbor

Laboratory

Press

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Methods

Parameter

Estimation

in

Biochemical

Pathways

A

Comparison

of

Global

Optimization

Methods

Carmen

G.

Moles

Pedro

Mendes

and

Julio

R.

Process

Engineering

Group

Instituto

de

Investigaciones

Marinas

-LRB-

CSIC

-RRB-

Vigo

Spain

Bioinformatics

Institute

Virginia

Polytechnic

Institute

and

State

University

Blacksburg

Virginia

USA

Here

we

address

the

problem

of

parameter

estimation

-LRB-

inverse

problem

-RRB-

of

nonlinear

dynamic

biochemical

pathways

This

problem

is

stated

as

a

nonlinear

programming

-LRB-

NLP

-RRB-

problem

subject

to

nonlinear

differential-algebraic

constraints

These

problems

are

known

to

be

frequently

ill-conditioned

and

multimodal

Thus

traditional

-LRB-

gradient-based

-RRB-

local

optimization

methods

fail

to

arrive

at

satisfactory

solutions

To

surmount

this

limitation

the

use

of

several

state-of-the-art

deterministic

and

stochastic

global

optimization

methods

is

explored

A

case

study

considering

the

estimation

of

parameters

of

a

nonlinear

biochemical

dynamic

model

is

taken

as

a

benchmark

Only

a

certain

type

of

stochastic

algorithm

evolution

strategies

-LRB-

ES

-RRB-

is

able

to

solve

this

problem

successfully

Although

these

stochastic

methods

can

not

guarantee

global

optimality

with

certainty

their

robustness

plus

the

fact

that

in

inverse

problems

they

have

a

known

lower

bound

for

the

cost

function

make

them

the

best

available

candidates

Mathematical

optimization

can

be

used

as

a

computational

engine

to

arrive

at

the

best

solution

for

a

given

problem

in

a

systematic

and

efficient

way

In

the

context

of

biochemical

systems

coupling

optimization

with

suitable

simulation

modules

opens

a

whole

new

avenue

of

possibilities

Mendes

and

Kell

-LRB-

-RRB-

highlight

two

types

of

important

applications

Design

problems

How

to

rationally

design

improved

metabolic

pathways

to

maximize

the

flux

of

interesting

products

and

minimize

the

production

of

undesired

by-products

-LRB-

metabolic

engineering

and

biochemical

evolution

studies

-RRB-

Parameter

estimation

Given

a

set

of

experimental

data

calibrate

the

model

so

as

to

reproduce

the

experimental

results

in

the

best

possible

way

This

contribution

considers

the

latter

case

that

is

the

socalled

inverse

problem

The

correct

solution

of

inverse

problems

plays

a

key

role

in

the

development

of

dynamic

models

which

in

turn

can

promote

functional

understanding

at

the

systems

level

as

shown

by

for

example

Swameye

et

al.

-LRB-

-RRB-

and

Cho

et

al.

-LRB-

-RRB-

for

signaling

pathways

The

paper

is

structured

as

follows

In

the

next

section

we

state

the

mathematical

problem

highlighting

its

main

characteristics

and

very

especially

its

challenging

nature

for

traditional

local

optimization

methods

Next

global

optimization

-LRB-

GO

-RRB-

is

presented

as

an

alternative

to

surmount

those

difficulties

A

brief

review

of

GO

methods

is

given

and

a

selection

of

the

presently

most

promising

alternatives

is

presented

The

following

section

outlines

a

case

study

considering

the

estimation

of

parameters

of

a

three-step

pathway

which

will

be

used

as

a

benchmark

to

compare

the

different

GO

methods

selected

A

Results

and

Discussion

section

follows

ending

with

a

set

of

Conclusions

ness

of

the

fit

of

the

model

with

respect

to

a

given

experimental

data

set

subject

to

the

dynamics

of

the

system

-LRB-

acting

as

a

set

of

differential

equality

constraints

-RRB-

plus

possibly

other

algebraic

constraints

Mathematically

the

formulation

is

that

of

a

nonlinear

programming

problem

-LRB-

NLP

-RRB-

with

differential-algebraic

constraints

Find

p

to

minimize

J

subject

to

f

dx

x

y

p

v

t

dt

x

h

x

y

p

v

g

x

y

p

v

pL

p

pU

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

tf

ymsd

t

y

p

t

T

W

t

ymsd

t

y

p

t

dt

-LRB-

-RRB-

METHODS

Statement

of

the

Inverse

Problem

Parameter

estimation

problems

of

nonlinear

dynamic

systems

are

stated

as

minimizing

a

cost

function

that

measures

the

Corresponding

author

E-MAIL

julio@iim.csic.es

FAX

Article

and

publication

are

at

gr

Article

published

online

before

print

in

October

where

J

is

the

cost

function

to

be

minimized

p

is

the

vector

of

decision

variables

of

the

optimization

problem

the

set

of

parameters

to

be

estimated

ymsd

is

the

experimental

measure

of

a

subset

of

the

-LRB-

so-called

-RRB-

output

state

variables

y

-LRB-

p

t

-RRB-

is

the

model

prediction

for

those

outputs

W

-LRB-

t

-RRB-

is

a

weighting

-LRB-

or

scaling

-RRB-

matrix

x

is

the

differential

state

variables

v

is

a

vector

of

other

-LRB-

usually

time-invariant

-RRB-

parameters

that

are

not

estimated

f

is

the

set

of

differential

and

algebraic

equality

constraints

describing

the

system

dynamics

-LRB-

i.e.

the

nonlinear

process

model

-RRB-

and

h

and

g

are

the

possible

equality

and

inequality

path

and

point

constraints

that

express

additional

requirements

for

the

system

performance

Finally

p

is

subject

to

upper

and

lower

bounds

acting

as

inequality

constraints

The

formulation

above

is

that

of

a

nonlinear

programming

problem

-LRB-

NLP

-RRB-

with

differential-algebraic

-LRB-

DAEs

-RRB-

constraints

Because

of

the

nonlinear

and

constrained

nature

of

the

system

dynamics

these

problems

are

very

often

multimodal

-LRB-

nonconvex

-RRB-

Therefore

if

this

NLP-DAEs

is

solved

via

standard

local

methods

such

as

the

standard

Levenberg-Marquardt

method

it

is

very

likely

that

the

solution

found

will

be

of

local

nature

as

discussed

by

Mendes

and

Kell

-LRB-

-RRB-

for

example

The

earliest

and

simplest

attempt

to

surmount

the

nonconvexity

of

many

optimization

problems

was

based

on

the

idea

of

using

a

local

method

repeatedly

starting

from

a

number

of

dif

by

Cold

Spring

Harbor

Laboratory

Press

ISSN

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Moles

et

al.

Global

Optimization

Methods

Global

optimization

methods

can

be

roughly

classified

as

deterministic

-LRB-

Horst

and

Tuy

Grossmann

Pinter

Esposito

and

Floudas

-RRB-

and

stochastic

strategies

-LRB-

Guus

et

al.

Ali

et

al.

Torn

et

al.

-RRB-

Stochastic

methods

for

global

optimization

ultimately

rely

on

probabilistic

approaches

Given

that

random

elements

are

involved

these

methods

only

have

weak

theoretical

guarantees

of

convergence

to

the

global

solution

Deterministic

methods

are

those

that

can

provide

a

level

of

assurance

that

the

global

optimum

will

be

located

and

several

important

advances

in

the

GO

of

certain

types

of

nonlinear

dynamic

systems

have

been

made

recently

-LRB-

Esposito

and

Floudas

Singer

et

al.

Papamichail

and

Adjiman

-RRB-

However

it

should

be

noted

that

although

deterministic

methods

can

guarantee

global

optimality

for

certain

GO

problems

no

algorithm

can

solve

general

GO

problems

with

certainty

in

finite

time

-LRB-

Guus

et

al.

-RRB-

In

fact

although

several

classes

of

deterministic

methods

-LRB-

e.g.

branch

and

bound

-RRB-

have

sound

theoretical

convergence

properties

the

associated

computational

effort

increases

very

rapidly

-LRB-

often

exponentially

-RRB-

with

the

problem

size

In

contrast

many

stochastic

methods

can

locate

the

vicinity

of

global

solutions

with

relative

efficiency

but

the

cost

to

pay

is

that

global

optimality

can

not

be

guaranteed

However

in

practice

the

user

can

be

satisfied

if

these

methods

provide

a

very

good

-LRB-

often

the

best

available

-RRB-

solution

in

modest

computation

times

Furthermore

stochastic

methods

are

usually

quite

simple

to

implement

and

use

and

they

do

not

require

transformation

of

the

original

problem

which

can

be

treated

as

a

black

box

This

characteristic

is

especially

interesting

because

very

often

the

researcher

must

link

the

optimizer

with

a

third-party

software

package

in

which

the

process

dynamic

model

has

been

implemented

Figure

The

model

metabolic

pathway

used

in

these

studies

Solid

arrows

represent

mass

flow

dashed

arrows

represent

kinetic

regulation

arrow

ends

represent

activation

blunt

ends

inhibition

S

and

P

are

the

pathway

substrate

and

product

and

are

held

at

constant

concentrations

and

are

intermediate

metabolites

of

the

pathway

and

are

the

enzymes

and

are

the

mRNA

species

for

the

enzymes

ferent

initial

decision

vectors

which

is

the

so-called

multistart

strategy

-LRB-

Guus

et

al.

-RRB-

However

the

approach

usually

does

not

work

for

realistic

applications

because

it

exhibits

a

major

drawback

When

many

starting

points

are

used

the

same

minimum

will

eventually

be

determined

several

times

-LRB-

Torn

Guus

et

al.

-RRB-

thus

the

method

becomes

very

inefficient

Clustering

methods

and

other

so-called

global

optimization

-LRB-

GO

-RRB-

methods

-LRB-

more

details

given

below

-RRB-

have

been

developed

to

ensure

better

efficiency

and

robustness

Mendes

and

Kell

-LRB-

-RRB-

considered

the

parameter

estimation

of

several

rate

constants

of

the

mechanism

of

irreversible

inhibition

of

HIV

proteinase

This

problem

has

a

total

of

parameters

to

estimate

and

these

authors

obtained

the

best

fit

using

the

simulated

annealing

-LRB-

SA

-RRB-

method

However

they

highlighted

the

huge

computational

effort

associated

with

this

method

noting

that

the

topic

deserves

more

detailed

study

More

recently

Mendes

-LRB-

-RRB-

considered

a

larger

inverse

problem

regarding

a

three-step

pathway

finding

that

gradient

methods

could

not

converge

to

the

solution

from

any

arbitrary

starting

vector

After

comparing

a

number

of

stochastic

methods

evolutionary

programming

-LRB-

EP

-RRB-

was

found

to

be

the

best

performing

algorithm

with

a

final

refined

solution

that

replicated

the

true

dynamics

rather

well

As

a

drawback

the

needed

computation

time

was

again

excessive

In

this

contribution

we

have

considered

this

same

threestep

pathway

as

a

benchmark

and

we

have

attempted

to

solve

the

associated

inverse

problem

using

several

state-of-the-art

deterministic

and

stochastic

global

optimization

-LRB-

GO

-RRB-

algorithms

Our

main

objective

was

to

investigate

if

the

present

state

of

GO

can

provide

us

with

a

more

efficient

and

reliable

method

for

this

class

of

problems

Stochastic

GO

Methods

There

are

many

different

kinds

of

stochastic

methods

for

global

optimization

but

the

following

groups

must

be

highlighted

Adaptive

stochastic

methods

-LRB-

or

adaptive

random

search

-RRB-

which

were

originally

developed

in

the

domains

of

electrical

and

control

engineering

and

applied

mathematics

in

the

and

-LRB-

Brooks

Matyas

Rastrigin

and

Rubinstein

-RRB-

Clustering

methods

-LRB-

Torn

Rinnooy-Kan

and

Timmer

-RRB-

were

derived

from

the

initial

concepts

of

multistart

methods

that

is

local

methods

started

from

different

initial

points

Clustering

methods

are

more

efficient

and

robust

than

multistart

methods

because

they

try

to

identify

the

vicinity

of

local

optima

thus

increasing

efficiency

by

avoiding

the

repeated

determination

of

the

same

local

solutions

However

they

do

not

seem

to

work

well

for

a

large

number

of

decision

variables

Evolutionary

computation

-LRB-

EC

-RRB-

also

known

as

biologically

inspired

methods

or

population-based

stochastic

methods

This

Table

Search

Parameters

Utilized

in

the

Different

Algorithms

SRES

Search

parameters

G

lambda

pf

varphi

ICRS

Search

parameters

e

kl

maxfsd

uES

G

lambda

varphi

MCS

n

smax

nf

stop

-LRB-

-RRB-

stop

-LRB-

-RRB-

n

iinit

local

DE

VTR

st

D

NP

iterMax

F

CR

GBLSOLVE

Iterations

CMA-ES

N

lambda

xlow

xl

xup

xu

MaxFunEvals

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Parameter

Estimation

in

Biochemical

Pathways

is

a

very

popular

class

of

methods

based

on

the

ideas

of

biological

evolution

-LRB-

Fogel

-RRB-

which

is

driven

by

the

mechanisms

of

reproduction

mutation

and

the

principle

of

survival

of

the

fittest

-LRB-

Darwin

-RRB-

Similarly

to

biological

evolution

evolutionary

computing

methods

generate

better

and

better

solutions

by

iteratively

creating

new

generations

by

means

of

those

mechanisms

in

numerical

form

EC

methods

are

usually

classified

into

three

groups

Genetic

Algorithms

-LRB-

GAs

Goldberg

Holland

Michalewicz

-RRB-

Evolutionary

Programming

-LRB-

EP

Fogel

et

al.

-RRB-

and

Evolution

Strategies

-LRB-

ES

Schwefel

Beyer

and

Schwefel

-RRB-

Simulated

annealing

-LRB-

SA

and

other

similar

physically

inspired

methods

-RRB-

is

another

extremely

popular

class

of

methods

These

methods

were

created

by

simulating

certain

natural

phenomena

taking

place

in

for

example

the

cooling

of

metals

where

atoms

adopt

the

most

stable

configuration

as

slow

cooling

of

a

metal

takes

place

-LRB-

Kirkpatrick

et

al.

van

Laarhoven

and

Aarts

-RRB-

Other

meta-heuristics

Several

stochastic

methods

have

been

presented

during

recent

years

that

are

mostly

based

on

other

biological

or

physical

phenomena

and

with

combinatorial

optimization

as

their

original

domain

of

application

Examples

of

these

more

recent

methods

are

Taboo

Search

-LRB-

TS

-RRB-

Ant

Colony

Optimization

-LRB-

ACO

-RRB-

and

particle

swarm

methods

A

thorough

review

of

these

and

other

recent

techniques

can

be

found

in

Corne

et

al.

-LRB-

-RRB-

Table

P

value

S

value

Experiment

Generation

Element

of

decision

variables

vector

Nominal

value

Parameter

GO

Methods

Used

In

this

study

we

have

considered

a

set

of

selected

stochastic

and

deterministic

GO

methods

that

can

handle

black-box

models

The

selection

has

been

made

based

on

their

published

performance

and

on

our

own

experiences

considering

their

results

for

a

set

of

GO

benchmark

problems

Although

of

these

methods

can

guarantee

optimality

at

least

the

researcher

can

solve

a

given

problem

with

different

methods

and

make

a

decision

based

on

the

set

of

solutions

found

Usually

several

of

the

methods

will

converge

to

essentially

the

same

-LRB-

best

-RRB-

solution

It

should

be

noted

that

although

this

result

can

not

be

regarded

as

a

confirmation

of

global

optimality

-LRB-

it

might

be

the

same

local

optimum

-RRB-

it

does

give

the

user

some

extra

confidence

Furthermore

it

is

usually

possible

to

have

estimates

of

lower

bounds

for

the

cost

function

and

its

different

terms

so

the

goodness

of

the

global

solution

can

be

evaluated

-LRB-

sometimes

a

good

enough

solution

is

sufficient

-RRB-

The

GO

methods

that

we

have

considered

are

GBLSOLVE

A

deterministic

GO

method

implemented

in

Matlab

as

part

of

the

optimization

environment

TOMLAB

-LRB-

Holmstrom

-RRB-

It

is

a

version

of

the

DIRECT

algorithm

-LRB-

Jones

et

al.

Jones

-RRB-

that

handles

nonlinear

and

integer

constraints

GBLSOLVE

runs

for

a

predefined

number

of

iterations

and

considers

the

best

function

value

found

as

the

global

optimum

MCS

The

Multilevel

Coordinate

Search

algorithm

by

Huyer

and

Neumaier

-LRB-

-RRB-

also

inspired

by

the

DIRECT

method

-LRB-

Jones

-RRB-

is

an

intermediate

between

purely

heuristic

methods

and

those

allowing

an

assessment

of

the

quality

of

the

minimum

obtained

It

has

an

initial

global

phase

after

which

a

local

procedure

based

on

an

SQP

algorithm

is

launched

These

local

enhancements

S

and

P

values

were

combined

to

generate

a

total

of

sets

of

pseudoexperimental

measurements

and

nominal

values

of

the

parameters

also

indicating

the

corresponding

element

of

the

decision

variables

vector

-LRB-

p

-RRB-

lead

to

quick

convergence

once

the

global

step

has

found

a

point

in

the

basin

of

attraction

of

a

global

minimizer

ICRS

An

adaptive

stochastic

presented

by

Banga

and

Casares

-LRB-

-RRB-

improving

the

Controlled

Random

Search

-LRB-

CRS

-RRB-

method

of

Goulcher

and

Casares

-LRB-

-RRB-

Basically

ICRS

is

a

sequential

-LRB-

one

trial

vector

at

a

time

-RRB-

adaptive

random

search

method

that

can

handle

inequality

constraints

via

penalty

functions

and

which

has

been

successfully

applied

to

a

number

of

dynamic

optimization

problems

-LRB-

Banga

and

Seider

Banga

et

al.

-RRB-

Table

Results

of

the

Global

Optimization

Methods

SRES

J

Neval

CPU

time

-LRB-

h

-RRB-

uES

DE

ICRS

CMA-ES

MCS

GBLSOLVE

Objective

function

J

number

of

function

evaluations

Neval

and

computation

time

in

hours

of

a

PC/Pentium

III

MHz

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Moles

et

al.

DE

The

Differential

Evolution

method

as

presented

by

Storn

and

Price

-LRB-

-RRB-

DE

is

a

heuristic

population-based

approach

to

GO

The

original

code

of

the

DE

algorithm

-LRB-

Storn

and

Price

-RRB-

did

not

check

if

the

new

generated

vectors

were

within

their

bound

constraints

therefore

we

have

slightly

modified

the

code

for

that

purpose

uES

The

unconstrained

Evolution

Strategy

-LRB-

uES

-RRB-

is

a

-LRB-

-RRB-

ES

evolutionary

optimization

algorithm

-LRB-

based

on

Schwefel

-RRB-

for

problems

only

constrained

by

bounds

on

the

decision

variables

SRES

The

Evolution

Strategy

using

Stochastic

Ranking

-LRB-

Runarsson

and

Yao

-RRB-

is

a

-LRB-

-RRB-

ES

evolutionary

optimization

algorithm

that

uses

stochastic

ranking

as

the

constraint

handling

technique

The

stochastic

ranking

is

based

on

the

bubble-sort

algorithm

and

is

supported

by

the

idea

of

dominance

It

adjusts

the

balance

between

the

objective

and

penalty

functions

automatically

during

the

evolutionary

search

CMA-ES

Another

Evolution

Strategy

method

To

improve

the

convergence

rates

especially

on

nonseparable

and/or

badly

scaled

functions

this

method

introduces

in

the

evolution

strategy

the

intermediate

-LRB-

center

of

mass

-RRB-

recombination

with

derandomized

covariance

matrix

adaptation

-LRB-

CMA

-RRB-

This

is

a

generalized

individual

step-size

control

approach

independent

of

the

orientation

and

permutation

of

the

coordinate

axes

-LRB-

Hansen

and

Ostermeier

-RRB-

Table

Decision

Vector

for

the

Best

Solution

-LRB-

Found

by

SRES

-RRB-

Elements

of

best

vector

Computation

methods

they

have

been

shown

to

exhibit

nice

properties

especially

regarding

their

good

scaling

with

the

problem

size

-LRB-

Zabinsky

and

Smith

-RRB-

In

fact

Evolutionary

Computation

methods

can

be

regarded

as

population-based

extensions

of

adaptive

random

search

methods

Evolutionary

Computation

-LRB-

EC

-RRB-

methods

The

rest

of

the

selected

methods

DE

uES

SRES

and

CMA-ES

have

been

included

as

the

most

competitive

representatives

of

EC

methods

DE

-LRB-

Storn

and

Price

-RRB-

can

be

considered

as

a

hybrid

between

adaptive

random

search

methods

and

genetic

algorithms

-LRB-

GAs

-RRB-

Although

pure

GAs

are

by

far

the

most

popular

EC

methods

an

increasing

amount

of

literature

during

recent

years

has

consistently

shown

that

ES

methods

closely

followed

by

EP

are

usually

more

efficient

and

robust

than

GAs

especially

for

continuous

problems

-LRB-

Hoffmeister

and

B

ck

Saravanan

et

al.

B

ck

Balsa-Canto

et

al.

-RRB-

For

this

reason

we

have

selected

SRES

and

CMA-ES

as

recent

and

competitive

ES

methods

Their

superiority

over

more

traditional

GAs

has

been

shown

considering

many

different

test

problems

-LRB-

Hansen

and

Ostermeier

Runarsson

and

Yao

-RRB-

It

should

be

noted

that

we

have

not

included

any

representative

of

the

type

of

GO

methods

known

as

Simulated

Annealing

-LRB-

SA

-RRB-

which

are

almost

as

popular

as

GAs

As

in

the

case

of

GAs

the

decision

to

exclude

SA-based

methods

was

based

on

its

reported

poor

performance

with

respect

to

the

above

selected

methods

It

should

be

noted

that

both

GAs

and

SA

were

originally

devised

for

combinatorial

optimization

problems

-LRB-

i.e.

those

with

discrete

decision

variables

-RRB-

and

later

adapted

for

global

optimization

in

real

valued

search

spaces

In

contrast

the

above

methods

were

designed

with

real

-LRB-

continuous

-RRB-

decision

Justification

of

the

Selection

The

above

set

of

GO

methods

should

be

regarded

as

a

balanced

selection

of

competitive

algorithms

trying

to

reflect

the

state

of

the

art

for

each

type

of

approach

Deterministic

methods

GBLSOLVE

was

chosen

as

a

recent

method

with

good

reported

results

for

several

challenging

problems

Similarly

the

related

and

similarly

recent

MCS

method

was

included

as

a

qualified

representative

of

those

methods

combining

deterministic

approaches

with

some

heuristics

Adaptive

stochastic

methods

The

ICRS

algorithm

the

oldest

in

the

set

was

taken

as

a

representative

of

the

adaptive

random

search

methods

Although

this

type

of

algorithm

is

not

as

popular

as

for

example

Evolutionary

Figure

Convergence

curves

-LRB-

objective

function

versus

computation

time

in

seconds

using

a

PC/Pentium

III

MHz

-RRB-

Figure

method

Histogram

of

the

results

obtained

with

the

multistart

local

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Parameter

Estimation

in

Biochemical

Pathways

dt

dt

dt

p

p

p

S

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

dt

dt

dt

dt

S

S

P

p

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Figure

predicted

-LRB-

continuous

line

-RRB-

and

experimental

-LRB-

marker

-RRB-

behavior

for

the

experiments

variables

in

mind

and

this

could

be

one

of

the

main

reasons

for

their

better

efficiency

and

robustness

for

this

class

of

problems

Implementation

Details

For

the

sake

of

fair

comparison

we

have

considered

Matlab

-LRB-

-RRB-

implementations

of

all

these

methods

The

main

reason

to

use

Matlab

is

that

it

is

a

convenient

environment

to

postprocess

and

visualize

all

the

information

arising

from

the

optimization

runs

of

the

different

solvers

allowing

careful

comparisons

with

little

programming

effort

Furthermore

new

methods

-LRB-

or

modifications

to

existing

ones

-RRB-

can

be

easily

prototyped

and

evaluated

However

as

a

drawback

it

is

well

known

that

Matlab

programs

usually

are

one

order

of

magnitude

-LRB-

or

more

-RRB-

slower

than

equivalent

compiled

Fortran

or

C

codes

To

minimize

this

effect

we

have

implemented

the

more

costly

part

of

the

problem

-LRB-

i.e.

system

dynamic

simulation

plus

objective

function

evaluation

-RRB-

in

a

compiled

module

using

LSODA

as

the

initial

value

solver

-LRB-

Hindmarsh

-RRB-

This

solver

is

able

to

solve

both

stiff

and

nonstiff

systems

with

automatic

switching

between

the

necessary

numerical

schemes

The

resulting

module

-LRB-

a

dynamic

link

library

-RRB-

is

callable

from

the

Matlab

solvers

via

suitable

gateways

Because

most

stochastic

methods

use

-LRB-

or

more

-RRB-

of

the

computation

time

in

system

simulations

-LRB-

especially

if

their

complexity

level

is

medium

to

large

-RRB-

this

procedure

ensures

good

efficiency

while

retaining

the

main

advantages

of

the

Matlab

environment

Table

shows

the

internal

search

parameters

used

for

the

different

algorithms

which

were

selected

according

to

either

published

recommendations

and/or

our

personal

experience

with

several

preliminary

runs

Finally

to

illustrate

the

comparative

performance

of

multistart

local

methods

for

this

type

of

problem

a

multistart

code

-LRB-

named

ms-FMINCON

-RRB-

was

also

implemented

in

Matlab

making

use

of

the

FMINCON

code

which

is

part

of

the

MATLAB

Optimization

Toolbox

-LRB-

Anonymous

-RRB-

FMINCON

is

a

gradientbased

solver

indicated

for

unconstrained

functions

Its

default

algorithm

is

a

quasi-Newtonian

method

that

uses

the

BFGS

formula

for

updating

the

approximation

of

the

Hessian

matrix

Its

default

line

search

algorithm

is

a

safeguarded

mixed

quadratic

and

cubic

polynomial

interpolation

and

extrapolation

method

-LRB-

-RRB-

dt

-LRB-

-RRB-

where

and

represent

the

concentrations

of

the

species

involved

in

the

different

biochemical

reactions

and

S

and

P

keep

fixed

initial

values

for

each

experiment

-LRB-

i.e.

parameters

under

our

control

-RRB-

The

optimization

problem

is

then

to

fit

the

remaining

parameters

which

are

divided

into

two

different

classes

Hill

coefficients

allowed

to

vary

within

the

Case

Study

A

Three-Step

Pathway

The

optimization

problem

consists

of

the

estimation

of

kinetic

parameters

of

a

nonlinear

biochemical

dynamic

model

formed

by

ODEs

that

describe

the

variation

of

the

metabolite

concentrations

with

time

-LRB-

see

Fig.

-RRB-

The

mathematical

formulation

of

the

nonlinear

dynamic

model

is

Figure

predicted

-LRB-

continuous

line

-RRB-

and

experimental

-LRB-

marker

-RRB-

behavior

for

the

experiments

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Moles

et

al.

Figure

Relative

error

-LRB-

-RRB-

for

the

estimated

parameters

-LRB-

for

the

best

solution

obtained

by

SRES

-RRB-

range

-LRB-

-RRB-

and

all

the

others

allowed

to

vary

within

the

range

-LRB-

-RRB-

The

global

optimization

problem

is

stated

as

the

minimization

of

a

weighted

distance

measure

J

between

experimental

and

predicted

values

of

the

state

variables

represented

by

the

vector

y

n

m

J

i

j

wij

ypred

i

yexp

i

j

-LRB-

-RRB-

where

n

is

the

number

of

data

for

each

experiment

m

is

the

number

of

experiments

yexp

represents

the

known

experimental

data

and

ypred

is

the

vector

of

states

that

corresponds

to

the

predicted

theoretical

evolution

using

the

model

with

a

given

set

of

the

parameters

Furthermore

wij

corresponds

to

the

different

weights

taken

to

normalize

the

contributions

of

each

term

-LRB-

i.e.

wij

-LCB-

-LSB-

yexp

-LRB-

i

-RRB-

-RSB-

j

-RCB-

-RRB-

To

better

assess

the

performance

of

the

GO

techniques

for

the

solution

of

the

inverse

problem

pseudoexperimental

data

were

generated

by

simulation

from

a

set

of

chosen

parameters

-LRB-

to

be

considered

as

the

true

or

nominal

values

-RRB-

Thus

pseudomeasurements

of

the

concentrations

of

metabolites

proteins

and

messenger

RNA

species

were

the

result

of

different

experiments

-LRB-

simulations

-RRB-

in

which

the

initial

concentrations

of

the

pathway

substrate

S

and

product

P

were

varied

-LRB-

see

Table

to

examine

S

and

P

values

for

each

experimental

design

plus

the

nominal

values

considered

for

the

parameters

-RRB-

These

simulated

data

represent

exact

results

that

is

devoid

of

measurement

noise

RESULTS

AND

DISCUSSION

All

the

computations

were

performed

using

a

PC/Pentium

III

-LRB-

MHz

-RRB-

platform

running

Windows

The

best

result

-LRB-

J

-RRB-

was

obtained

using

the

SRES

method

-LRB-

Runarsson

and

Yao

-RRB-

after

a

total

computation

time

of

h.

However

it

should

be

noted

that

this

large

computational

effort

was

the

consequence

of

the

very

tight

convergence

criteria

used

but

from

a

practical

point

of

view

an

almost

equally

good

result

was

reached

in

a

few

hours

The

second

best

method

was

the

uES

method

which

converged

to

a

very

close

value

of

J

Thus

uES

can

be

regarded

as

a

very

close

second

winner

Detailed

results

are

given

in

Table

with

the

best

solution

vector

shown

in

Table

Table

shows

that

of

the

other

algorithms

tested

neither

deterministic

nor

stochastic

could

arrive

at

the

vicinity

of

the

abovementioned

solutions

This

is

indeed

a

clear

sign

of

the

very

challenging

nature

of

these

problems

The

results

in

Table

also

reveal

an

interesting

result

that

might

be

surprising

at

first

sight

The

MCS

and

GBLSOLVE

methods

present

a

larger

computation

time

per

function

evaluation

than

the

rest

of

the

algorithms

However

this

is

simply

a

consequence

of

the

larger

overhead

introduced

by

these

methods

to

generate

each

new

decision

vector

Simply

comparing

the

final

cost

function

values

and

the

overall

computation

times

can

be

misleading

In

Figure

the

convergence

curves

-LRB-

objective

function

versus

computation

time

-RRB-

of

the

best

five

methods

all

of

them

stochastic

are

plotted

-LRB-

note

the

log

log

scales

-RRB-

It

is

quite

clear

that

SRES

and

uES

presented

the

best

convergence

rates

at

all

times

It

is

indeed

surprising

to

note

that

several

global

optimization

methods

-LRB-

e.g.

DE

ICRS

-RRB-

which

in

the

past

presented

a

very

good

performance

dealing

with

a

number

of

hard

GO

problems

have

clearly

failed

here

This

is

probably

because

of

the

greater

complexity

of

this

parameter

estimation

problem

-LRB-

i.e.

a

very

large

number

of

local

solutions

-RRB-

and/or

its

relatively

large

dimensionality

In

the

case

of

ICRS

its

reported

CPU

time

in

Table

is

much

shorter

that

those

of

the

other

algorithms

but

this

is

only

a

consequence

of

its

early

stop

at

a

local

minimum

In

fact

the

true

story

about

comparative

performance

is

told

in

Figure

-LRB-

convergence

curves

-RRB-

where

it

can

be

seen

that

the

uES

and

SRES

methods

were

better

than

ICRS

at

all

times

Moreover

by

the

time

ICRS

stopped

-LRB-

h

-RRB-

the

running

ES-based

methods

had

already

arrived

at

much

better

objective

function

values

as

shown

in

Figure

In

the

case

of

the

MCS

and

GBLSOLVE

methods

which

are

both

inspired

by

the

DIRECT

method

of

Jones

-LRB-

-RRB-

the

results

are

probably

due

to

the

dimensionality

of

the

problem

but

also

to

implementation

issues

at

least

in

the

case

of

GBLSOLVE

because

more

efficient

implementations

have

been

reported

in

the

latest

version

of

the

TOMLAB

library

-LRB-

Holmstrom

-RRB-

Regarding

the

multistart

local

search

the

ms-FMINCON

method

was

executed

starting

from

random

initial

points

The

best

result

was

J

very

far

from

the

best

solution

obtained

with

SRES

Furthermore

most

of

the

other

local

solutions

obtained

were

much

worse

Figure

shows

a

histogram

with

the

distribution

of

all

the

local

solutions

found

with

this

method

Clearly

and

in

contrast

with

popular

belief

-LRB-

or

folklore

in

the

words

of

Guus

et

al.

-RRB-

multistart

local

methods

are

unsatisfactory

for

hard

parameter

estimation

problems

like

this

one

Figures

and

show

a

comparison

-LRB-

between

the

predicted

and

experimental

data

-RRB-

for

the

best

decision

vector

-LRB-

found

with

SRES

-RRB-

of

the

concentrations

and

It

is

worth

noting

the

very

good

correlation

between

the

experimental

and

predicted

data

The

representation

of

the

dynamic

behavior

for

the

other

variables

is

quite

similar

and

is

not

included

here

for

the

sake

of

brevity

Finally

to

underline

the

merit

of

the

best

solution

obtained

by

the

SRES

optimizer

the

error

values

that

is

the

relative

differences

between

the

known

-LRB-

real

-RRB-

and

the

estimated

parameters

are

shown

in

Figure

Only

parameters

and

were

estimated

with

an

error

All

the

other

errors

were

below

with

the

majority

being

below

which

is

a

very

remarkable

result

The

earlier

study

had

not

been

able

to

achieve

such

good

results

-LRB-

Mendes

-RRB-

We

believe

that

this

is

because

here

we

used

the

results

of

different

experiments

rather

than

just

a

single

one

The

data

used

in

the

earlier

study

reflected

a

greater

level

of

underdetermination

and

this

resulted

in

the

somewhat

imperfect

solution

obtained

By

covering

a

larger

set

of

dynamics

that

the

model

is

capable

of

we

have

been

able

to

obtain

near-exact

results

-LRB-

convergence

to

the

global

optimum

-RRB-

We

stress

that

these

excellent

results

were

obtained

-LRB-

as

before

-RRB-

in

conditions

in

which

measurement

noise

is

absent

whereas

in

reality

one

expects

it

to

be

present

to

some

extent

sometimes

even

in

considerable

mag

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Parameter

Estimation

in

Biochemical

Pathways

nitudes

-LRB-

e.g.

microarray

data

-RRB-

Nevertheless

the

problem

was

already

sufficiently

hard

that

most

algorithms

did

not

perform

satisfactorily

Later

we

intend

to

repeat

this

study

in

the

presence

of

measurement

noise

Conclusions

Only

evolutionary

strategies

namely

the

SRES

and

uES

methods

were

able

to

successfully

solve

the

inverse

problem

associated

with

a

three-step

pathway

This

result

is

in

agreement

with

other

recent

global

optimization

studies

-LRB-

Rechenberg

Runarsson

and

Yao

Costa

and

Oliveira

-RRB-

some

of

them

dealing

with

nonlinear

dynamic

systems

-LRB-

Moles

et

al.

Banga

et

al.

-RRB-

which

indicate

that

evolution

strategies

might

be

the

most

competitive

stochastic

optimization

method

especially

for

large

problems

A

possible

drawback

of

ES

methods

in

spite

of

these

good

results

is

the

computational

effort

required

which

is

on

the

order

of

hours

using

low-cost

PC

platforms

However

it

is

well

known

that

many

stochastic

methods

including

ES

lend

themselves

to

parallelization

very

easily

which

means

that

this

problem

could

be

handled

in

reasonable

wallclock

time

by

suitable

parallel

versions

Present

technologies

for

cluster

computing

-LRB-

e.g.

-RRB-

or

grid

computing

-LRB-

e.g.

http

-RRB-

can

greatly

facilitate

the

development

of

such

versions

One

of

us

-LRB-

P.M.

-RRB-

wrote

and

distributes

a

biochemical

simulation

package

Gepasi

-LRB-

Mendes

-RRB-

which

implements

several

numerical

optimization

methods

including

GO

Although

that

software

is

easily

extensible

most

of

the

methods

implemented

and

tested

here

are

not

yet

included

in

that

software

Given

the

superior

performance

of

SRES

and

uES

these

are

now

being

implemented

in

that

package

which

will

make

them

available

to

the

biochemical

modeling

community

at

large

ACKNOWLEDGMENTS

P.M.

thanks

the

National

Science

Foundation

-LRB-

grants

and

-RRB-

the

Commonwealth

of

Virginia

and

Sun

Microsystems

for

financial

support

J.R.B.

thanks

the

Spanish

Government

-LRB-

MCyT

project

-RRB-

for

financial

support

The

publication

costs

of

this

article

were

defrayed

in

part

by

payment

of

page

charges

This

article

must

therefore

be

hereby

marked

advertisement

in

accordance

with

USC

section

solely

to

indicate

this

fact

REFERENCES

Ali

M.M.

Storey

C.

and

Torn

A.

Application

of

stochastic

global

optimization

algorithms

to

practical

problems

J.

Optim

Theory

Appl

Anonymous

Optimization

toolbox

user

's

guide

The

Math

Works

Inc.

Natick

MA

B

ck

T.

Evolution

strategies

An

alternative

evolutionary

algorithm

In

Artificial

evolution

-LRB-

eds

J.M.

Alliott

et

al.

-RRB-

pp.

Springer

Berlin

Balsa-Canto

E.

Alonso

A.A.

and

Banga

J.R.

Dynamic

optimization

of

bioprocesses

Deterministic

and

stochastic

strategies

In

Proceedings

of

ACoFop

IV

-LRB-

Automatic

Control

of

Food

and

Biological

Processes

-RRB-

-LRB-

eds

C.

Skjoldebremd

and

G.

Trystrom

-RRB-

pp.

Goteborg

Sweden

Banga

J.R.

and

Casares

J.J.

ICRS

Application

to

a

wastewater

treatment

plant

model

In

IChemE

Symposium

Series

No.

pp.

Pergamon

Press

Oxford

UK

Banga

J.R.

and

Seider

W.D.

Global

optimization

of

chemical

processes

using

stochastic

algorithms

In

State

of

the

art

in

global

optimization

-LRB-

eds

C.A.

Floudas

and

P.M.

Pardalos

-RRB-

pp.

Kluwer

Academic

Publishers

Dordrecht

The

Netherlands

Banga

J.R.

Alonso

A.A.

and

Singh

R.P.

Stochastic

dynamic

optimization

of

batch

and

semicontinuous

bioprocesses

Biotechnology

Prog

Banga

J.R.

Alonso

A.A.

Moles

C.G.

and

Balsa-Canto

E.

Efficient

and

robust

numerical

strategies

for

the

optimal

control

of

non-linear

bio-processes

In

Proceedings

of

the

Mediterranean

Conference

on

Control

and

Automation

-LRB-

-RRB-

-LRB-

eds

J.S.

Sentievio

and

M.

Attrans

-RRB-

CD-ROM

IEEE

CSS

Lisbon

Portugal

Beyer

H.G.

and

Schwefel

H.P.

Evolution

strategies

A

comprehensive

introduction

Natural

Computing

Brooks

S.H.

A

discussion

of

random

methods

for

seeking

maxima

Op

Res

Cho

K.H.

Shin

S.Y.

Kim

H.W.

Wolkenhauer

O.

McFerran

B.

and

Kolch

W.

Mathematical

modelling

of

the

influence

of

RKIP

on

the

ERK

signalling

pathway

In

Computational

methods

in

systems

biology

-LRB-

ed

C.

Priami

-RRB-

pp.

Lecture

Notes

in

Computer

Science

-LRB-

LNCS

-RRB-

Springer

Verlag

New

York

Corne

D.

Dorigo

M.

and

Glover

F.

New

ideas

in

optimization

McGraw-Hill

New

York

Costa

L.

and

Oliveira

P.

Evolutionary

algorithms

approach

to

the

solution

of

mixed

integer

non-linear

programming

problems

Comput

Chem

Eng

Darwin

C.R.

On

the

origin

of

species

by

means

of

natural

selection

J.

Murray

London

Esposito

W.R.

and

Floudas

C.A.

Global

optimization

for

the

parameter

estimation

of

differential-algebraic

systems

Indust

Eng

Chem

Res

Fogel

D.B.

Evolutionary

computation

Toward

a

new

philosophy

of

machine

intelligence

IEEE

Press

New

York

Fogel

L.J.

Owens

A.J.

and

Walsh

M.J.

Artificial

intelligence

through

simulated

evolution

Wiley

New

York

Goldberg

D.E.

Genetic

algorithms

in

search

optimization

and

machine

learning

Addison

Wesley

Longman

London

Goulcher

R.

and

Casares

J.J.

The

solution

of

steady-state

chemical

engineering

optimisation

problems

using

a

random-search

algorithm

Comput

Chem

Eng

Grossmann

I.E.

Global

optimization

in

engineering

design

Kluwer

Academic

Publishers

Dordrecht

The

Netherlands

Guus

C.

Boender

E.

and

Romeijn

H.E.

Stochastic

methods

In

Handbook

of

global

optimization

-LRB-

eds

R.

Horst

and

P.M.

Pardalos

-RRB-

pp.

Kluwer

Academic

Publishers

Dordrecht

The

Netherlands

Hansen

N.

and

Ostermeier

A.

Convergence

properties

of

evolution

strategies

with

the

derandomized

covariance

matrix

adaptation

the

-LRB-

I

-RRB-

CMA-ES

In

Proceedings

of

the

European

Congress

on

Intelligent

Techniques

and

Soft

Computing

EUFIT

pp.

Verlag

Maing

Auchen

Germany

Hindmarsh

A.C.

ODEPACK

a

systematized

collection

of

ODE

solvers

In

Scientific

computing

-LRB-

ed

R.S.

Stepleman

-RRB-

pp.

North-Holland

Amsterdam

Hoffmeister

F.

and

B

ck

T.

Genetic

algorithms

and

evolution

strategies

Similarities

and

differences

In

Proceedings

of

Parallel

Problem

Solving

from

Nature

Workshop

PPSN

-LRB-

eds

H.-P

Schwefel

and

R.

M

nner

-RRB-

pp.

Lecture

Notes

in

Computer

Science

Vol

Springer-Verlag

Berlin

Holland

J.H.

Adaptation

in

natural

and

artificial

systems

An

introductory

analysis

with

applications

to

biology

control

and

artificial

intelligence

MIT

Press

Cambridge

MA

Holmstrom

K.

The

TOMLAB

optimization

environment

in

Matlab

Adv.

Model

Optim

Practical

optimization

with

the

TOMLAB

environment

in

Matlab

In

Proceedings

of

the

SIMS

Conference

pp.

Telemark

University

College

Porsgrunn

Norway

Horst

R.

and

Tuy

H.

Global

optimization

Deterministic

approaches

Springer-Verlag

Berlin

Huyer

W.

and

Neumaier

A.

A

global

optimization

by

multilevel

coordinate

search

J.

Global

Optim

Jones

D.R.

DIRECT

global

optimization

algorithm

In

Encyclopedia

of

optimization

-LRB-

eds

C.A.

Floudas

and

P.M.

Pardalos

-RRB-

pp.

Kluwer

Academic

Publishers

Dordrecht

The

Netherlands

Jones

D.R.

Perttunen

C.D.

and

Stuckman

B.E.

Lipschitzian

optimization

without

the

Lipschitz

constant

J.

Optimization

Theory

Appl

Kirkpatrick

S.

Gellatt

C.D.

and

Vecchi

M.P.

Optimization

by

simulated

annealing

Science

Matyas

J.

Random

optimization

Automat

Remote

Control

Mendes

P.

GEPASI

A

software

package

for

modelling

the

dynamics

steady

states

and

control

of

biochemical

and

other

systems

Comput

Appl

Biosci

Modeling

large

biological

systems

from

functional

genomic

data

Parameter

estimation

In

Foundations

of

systems

biology

-LRB-

ed

H.

Kitano

-RRB-

pp.

MIT

Press

Cambridge

MA

Mendes

P.

and

Kell

D.B.

Non-linear

optimization

of

biochemical

pathways

Applications

to

metabolic

engineering

and

parameter

estimation

Bioinformatics

Genome

Research

Downloaded

from

on

May

Published

by

Cold

Spring

Harbor

Laboratory

Press

Moles

et

al.

Michalewicz

Z.

Genetic

algorithms

data

structures

evolution

programs

Springer-Verlag

Berlin

New

York

Moles

C.G.

Gutierrez

G.

Alonso

A.A.

and

Banga

J.R.

Integrated

process

design

and

control

via

global

optimization

A

wastewater

treatment

plant

case

study

In

Proceedings

of

the

European

Control

Conference

-LRB-

ECC

-RRB-

September

-LRB-

ed

J.L.

Martins

-RRB-

CD-ROM

EUCA

Porto

Portugal

Papamichail

I.

and

Adjiman

C.S.

A

rigorous

global

optimization

algorithm

for

problems

with

ordinary

differential

equations

J.

Global

Optim

Pinter

J.

Global

optimization

in

action

Continuous

and

Lipschitz

optimization

Algorithms

implementations

and

applications

Kluwer

Academics

Publishers

Dordrecht

The

Netherlands

Rastrigin

L.A.

and

Rubinstein

Y.

The

comparison

of

random

search

and

stochastic

approximation

while

solving

the

problem

of

optimization

Automat

Control

Rechenberg

I.

Case

studies

in

evolutionary

experimentation

and

computation

Computer

Meth

Appl

Mech

Eng

Rinnooy-Kan

A.H.G.

and

Timmer

G.T.

Stochastic

global

optimization

methods

Part

I

Clustering

methods

Math

Prog

Runarsson

T.P.

and

Yao

X.

Stochastic

ranking

for

constrained

evolutionary

optimization

IEEE

Trans

Evol

Comput

Saravanan

N.

Fogel

D.B.

and

Nelson

K.M.

A

comparison

of

methods

for

self-adaptation

in

evolutionary

algorithms

Biosystems

Schwefel

H.P.

Evolution

and

optimum

seeking

Wiley

New

York

Singer

A.B.

Bok

J.K.

and

Barton

P.I.

Convex

underestimators

for

variational

and

optimal

control

problems

Comput

Aided

Chem

Eng

Storn

R.

and

Price

K.

Differential

Evolution

A

simple

and

efficient

heuristic

for

global

optimization

over

continuous

spaces

J.

Global

Optim

Swameye

I.

Muller

T.G.

Timmer

J.

Sandra

O.

and

Klingmuller

U.

Identification

of

nucleocytoplasmic

cycling

as

a

remote

sensor

in

cellular

signaling

by

databased

modeling

Proc

Natl.

Acad

Sci

Torn

A.A.

Global

optimization

as

a

combination

of

global

and

local

search

Proceedings

of

computer

simulation

versus

analytical

solutions

for

business

and

economic

models

pp.

Gothenburg

Sweden

Torn

A.

Ali

M.

and

Viitanen

S.

Stochastic

global

optimization

Problem

classes

and

solution

techniques

J.

Global

Opt

van

Laarhoven

P.J.M.

and

Aarts

E.H.L.

Simulated

annealing

Theory

and

applications

Reidel

Dordrecht

The

Netherlands

Zabinsky

Z.B.

and

Smith

R.L.

Pure

adaptive

search

in

global

optimization

Math

Prog

WEB

SITE

REFERENCES

technology

for

cluster

computing

Gepasi

biochemical

simulation

package

technology

for

grid

computing

Matlab

Received

February

accepted

in

revised

form

July

Genome

Research

O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Reducing

the

Dimensionality

of

Data

with

Neural

Networks

G.

E.

Hinton

et

al.

Science

-LRB-

-RRB-

DOI

science

The

following

resources

related

to

this

article

are

available

online

at

-LRB-

this

information

is

current

as

of

May

-RRB-

Updated

information

and

services

including

high-resolution

figures

can

be

found

in

the

online

version

of

this

article

at

Supporting

Online

Material

can

be

found

at

A

list

of

selected

additional

articles

on

the

Science

Web

sites

related

to

this

article

can

be

found

at

This

article

cites

articles

of

which

can

be

accessed

for

free

This

article

has

been

cited

by

article

-LRB-

s

-RRB-

on

the

ISI

Web

of

Science

This

article

has

been

cited

by

articles

hosted

by

HighWire

Press

see

Information

about

obtaining

reprints

of

this

article

or

about

obtaining

permission

to

reproduce

this

article

in

whole

or

in

part

can

be

found

at

Science

-LRB-

print

ISSN

online

ISSN

-RRB-

is

published

weekly

except

the

last

week

in

December

by

the

American

Association

for

the

Advancement

of

Science

New

York

Avenue

NW

Washington

DC

Copyright

by

the

American

Association

for

the

Advancement

of

Science

all

rights

reserved

The

title

Science

is

a

registered

trademark

of

AAAS

Downloaded

from

on

May

REPORTS

larized

nearly

vertically

For

completeness

Fig.

shows

the

off-resonant

case

for

the

smaller

SRRs

for

vertical

incident

polarization

Although

these

results

are

compatible

with

the

known

selection

rules

of

surface

SHG

from

usual

nonlinear

optics

-LRB-

-RRB-

these

selection

rules

do

not

explain

the

mechanism

of

SHG

Following

our

above

argumentation

on

the

magnetic

component

of

the

Lorentz

force

we

numerically

calculate

first

the

linear

electric

and

magnetic

field

distributions

-LRB-

-RRB-

from

these

fields

we

compute

the

electron

velocities

and

the

Lorentz-force

field

-LRB-

fig

-RRB-

In

the

spirit

of

a

metamaterial

the

transverse

component

of

the

Lorentz-force

field

can

be

spatially

averaged

over

the

volume

of

the

unit

cell

of

size

a

by

a

by

t

This

procedure

delivers

the

driving

force

for

the

transverse

SHG

polarization

As

usual

the

SHG

intensity

is

proportional

to

the

square

modulus

of

the

nonlinear

electron

displacement

Thus

the

SHG

strength

is

expected

to

be

proportional

to

the

square

modulus

of

the

driving

force

and

the

SHG

polarization

is

directed

along

the

driving-force

vector

Corresponding

results

are

summarized

in

Fig.

in

the

same

arrangement

as

Fig.

to

allow

for

a

direct

comparison

between

experiment

and

theory

The

agreement

is

generally

good

both

for

linear

optics

and

for

SHG

In

particular

we

find

a

much

larger

SHG

signal

for

excitation

of

those

two

resonances

-LRB-

Fig.

A

and

C

-RRB-

which

are

related

to

a

finite

magnetic-dipole

moment

-LRB-

perpendicular

to

the

SRR

plane

-RRB-

as

compared

with

the

purely

electric

Mie

resonance

-LRB-

Figs.

and

-RRB-

despite

the

fact

that

its

oscillator

strength

in

the

linear

spectrum

is

comparable

The

SHG

polarization

in

the

theory

is

strictly

vertical

for

all

resonances

Quantitative

deviations

between

the

SHG

signal

strengths

of

experiment

and

theory

respectively

are

probably

due

to

the

simplified

SRR

shape

assumed

in

our

calculations

and/or

due

to

the

simplicity

of

our

modeling

A

systematic

microscopic

theory

of

the

nonlinear

optical

properties

of

metallic

metamaterials

would

be

highly

desirable

but

is

currently

not

available

References

and

Notes

J.

B.

Pendry

A.

J.

Holden

D.

J.

Robbins

W.

J.

Stewart

IEEE

Trans

Microw

Theory

Tech

-LRB-

-RRB-

J.

B.

Pendry

Phys

Rev.

Lett

-LRB-

-RRB-

R.

A.

Shelby

D.

R.

Smith

S.

Schultz

Science

-LRB-

-RRB-

T.

J.

Yen

et

al.

Science

-LRB-

-RRB-

S.

Linden

et

al.

Science

-LRB-

-RRB-

C.

Enkrich

et

al.

Phys

Rev.

Lett

-LRB-

-RRB-

A.

N.

Grigorenko

et

al.

Nature

-LRB-

-RRB-

G.

Dolling

M.

Wegener

S.

Linden

C.

Hormann

Opt

Express

-LRB-

-RRB-

G.

Dolling

C.

Enkrich

M.

Wegener

C.

M.

Soukoulis

S.

Linden

Science

-LRB-

-RRB-

J.

B.

Pendry

D.

Schurig

D.

R.

Smith

Science

published

online

May

U.

Leonhardt

Science

-LRB-

-RRB-

published

online

May

M.

W.

Klein

C.

Enkrich

M.

Wegener

C.

M.

Soukoulis

S.

Linden

Opt

Lett

-LRB-

-RRB-

W.

J.

Padilla

A.

J.

Taylor

C.

Highstrete

M.

Lee

R.

D.

Averitt

Phys

Rev.

Lett

-LRB-

-RRB-

D.

R.

Smith

S.

Schultz

P.

Markos

C.

M.

Soukoulis

Phys

Rev.

B

-LRB-

-RRB-

S.

O'Brien

D.

McPeake

S.

A.

Ramakrishna

J.

B.

Pendry

Phys

Rev.

B

-LRB-

-RRB-

J.

Zhou

et

al.

Phys

Rev.

Lett

-LRB-

-RRB-

A.

K.

Popov

V.

M.

Shalaev

available

at

-LRB-

-RRB-

V.

G.

Veselago

Sov

Phys

Usp

-LRB-

-RRB-

M.

Wegener

Extreme

Nonlinear

Optics

-LRB-

Springer

Berlin

-RRB-

H.

M.

Barlow

Nature

-LRB-

-RRB-

S.-Y

Chen

M.

Maksimchuk

D.

Umstadter

Nature

-LRB-

-RRB-

Materials

and

Methods

are

available

as

supporting

material

on

Science

Online

P.

Guyot-Sionnest

W.

Chen

Y.

R.

Shen

Phys

Rev.

B

-LRB-

-RRB-

We

thank

the

groups

of

S.

W.

Koch

J.

V.

Moloney

and

C.

M.

Soukoulis

for

discussions

The

research

of

M.W.

is

supported

by

the

Leibniz

award

of

the

Deutsche

Forschungsgemeinschaft

-LRB-

DFG

-RRB-

that

of

S.L.

through

a

Helmholtz-Hochschul-Nachwuchsgruppe

-LRB-

-RRB-

Fig.

Theory

presented

as

the

experiment

-LRB-

see

Fig.

-RRB-

The

SHG

source

is

the

magnetic

component

of

the

Lorentz

force

on

metal

electrons

in

the

SRRs

The

setup

for

measuring

the

SHG

is

described

in

the

supporting

online

material

-LRB-

-RRB-

We

expect

that

the

SHG

strongly

depends

on

the

resonance

that

is

excited

Obviously

the

incident

polarization

and

the

detuning

of

the

laser

wavelength

from

the

resonance

are

of

particular

interest

One

possibility

for

controlling

the

detuning

is

to

change

the

laser

wavelength

for

a

given

sample

which

is

difficult

because

of

the

extremely

broad

tuning

range

required

Thus

we

follow

an

alternative

route

lithographic

tuning

-LRB-

in

which

the

incident

laser

wavelength

of

mm

as

well

as

the

detection

system

remains

fixed

-RRB-

and

tune

the

resonance

positions

by

changing

the

SRR

size

In

this

manner

we

can

also

guarantee

that

the

optical

properties

of

the

SRR

constituent

materials

are

identical

for

all

configurations

The

blue

bars

in

Fig.

summarize

the

measured

SHG

signals

For

excitation

of

the

LC

resonance

in

Fig.

-LRB-

horizontal

incident

polarization

-RRB-

we

find

an

SHG

signal

that

is

times

above

the

noise

level

As

expected

for

SHG

this

signal

closely

scales

with

the

square

of

the

incident

power

-LRB-

Fig.

-RRB-

The

polarization

of

the

SHG

emission

is

nearly

vertical

-LRB-

Fig.

-RRB-

The

small

angle

with

respect

to

the

vertical

is

due

to

deviations

from

perfect

mirror

symmetry

of

the

SRRs

-LRB-

see

electron

micrographs

in

Fig.

-RRB-

Small

detuning

of

the

LC

resonance

toward

smaller

wavelength

-LRB-

i.e.

to

wavelength

-RRB-

reduces

the

SHG

signal

strength

from

to

For

excitation

of

the

Mie

resonance

with

vertical

incident

polarization

in

Fig.

we

find

a

small

signal

just

above

the

noise

level

For

excitation

of

the

Mie

resonance

with

horizontal

incident

polarization

in

Fig.

a

small

but

significant

SHG

emission

is

found

which

is

again

po

Supporting

Online

Material

Materials

and

Methods

Figs

and

References

April

accepted

June

science

Reducing

the

Dimensionality

of

Data

with

Neural

Networks

G.

E.

Hinton

and

R.

R.

Salakhutdinov

High-dimensional

data

can

be

converted

to

low-dimensional

codes

by

training

a

multilayer

neural

network

with

a

small

central

layer

to

reconstruct

high-dimensional

input

vectors

Gradient

descent

can

be

used

for

fine-tuning

the

weights

in

such

autoencoder

networks

but

this

works

well

only

if

the

initial

weights

are

close

to

a

good

solution

We

describe

an

effective

way

of

initializing

the

weights

that

allows

deep

autoencoder

networks

to

learn

low-dimensional

codes

that

work

much

better

than

principal

components

analysis

as

a

tool

to

reduce

the

dimensionality

of

data

imensionality

reduction

facilitates

the

classification

visualization

communication

and

storage

of

high-dimensional

data

A

simple

and

widely

used

method

is

principal

components

analysis

-LRB-

PCA

-RRB-

which

D

finds

the

directions

of

greatest

variance

in

the

data

set

and

represents

each

data

point

by

its

coordinates

along

each

of

these

directions

We

describe

a

nonlinear

generalization

of

PCA

that

uses

an

adaptive

multilayer

Bencoder

-LSB-

network

JULY

VOL

SCIENCE

Downloaded

from

on

May

REPORTS

to

transform

the

high-dimensional

data

into

a

low-dimensional

code

and

a

similar

Bdecoder

-LSB-

network

to

recover

the

data

from

the

code

Department

of

Computer

Science

University

of

Toronto

King

's

College

Road

Toronto

Ontario

Canada

To

whom

correspondence

should

be

addressed

E-mail

hinton@cs.toronto.edu

Starting

with

random

weights

in

the

two

networks

they

can

be

trained

together

by

minimizing

the

discrepancy

between

the

original

data

and

its

reconstruction

The

required

gradients

are

easily

obtained

by

using

the

chain

rule

to

backpropagate

error

derivatives

first

through

the

decoder

network

and

then

through

the

encoder

network

-LRB-

-RRB-

The

whole

system

is

Decoder

Top

RBM

T

W

T

T

W

T

RBM

T

W

T

T

W

T

RBM

Code

layer

W

W

W

W

iZpixels

j

RBM

Encoder

X

i

j

jZfeatures

vi

hj

wij

Pretraining

Unrolling

Fine-tuning

Fig.

Pretraining

consists

of

learning

a

stack

of

restricted

Boltzmann

machines

-LRB-

RBMs

-RRB-

each

having

only

one

layer

of

feature

detectors

The

learned

feature

activations

of

one

RBM

are

used

as

the

data

for

training

the

next

RBM

in

the

stack

After

the

pretraining

the

RBMs

are

unrolled

to

create

a

deep

autoencoder

which

is

then

fine-tuned

using

backpropagation

of

error

derivatives

where

vi

and

hj

are

the

binary

states

of

pixel

i

and

feature

j

bi

and

bj

are

their

biases

and

wij

is

the

weight

between

them

The

network

assigns

a

probability

to

every

possible

image

via

this

energy

function

as

explained

in

-LRB-

-RRB-

The

probability

of

a

training

image

can

be

raised

by

Fig.

-LRB-

A

-RRB-

Top

to

bottom

Random

samples

of

curves

from

the

test

data

set

reconstructions

produced

by

the

six-dimensional

deep

autoencoder

reconstructions

by

logistic

PCA

-LRB-

-RRB-

using

six

components

reconstructions

by

logistic

PCA

and

standard

PCA

using

components

The

average

squared

error

per

image

for

the

last

four

rows

is

-LRB-

B

-RRB-

Top

to

bottom

A

random

test

image

from

each

class

reconstructions

by

the

autoencoder

reconstructions

by

logistic

PCA

and

standard

PCA

The

average

squared

errors

for

the

last

three

rows

are

and

-LRB-

C

-RRB-

Top

to

bottom

Random

samples

from

the

test

data

set

reconstructions

by

the

autoencoder

reconstructions

by

PCA

The

average

squared

errors

are

and

SCIENCE

VOL

JULY

Downloaded

from

on

May

called

an

Bautoencoder

-LSB-

and

is

depicted

in

Fig.

It

is

difficult

to

optimize

the

weights

in

nonlinear

autoencoders

that

have

multiple

hidden

layers

-LRB-

-RRB-

With

large

initial

weights

autoencoders

typically

find

poor

local

minima

with

small

initial

weights

the

gradients

in

the

early

layers

are

tiny

making

it

infeasible

to

train

autoencoders

with

many

hidden

layers

If

the

initial

weights

are

close

to

a

good

solution

gradient

descent

works

well

but

finding

such

initial

weights

requires

a

very

different

type

of

algorithm

that

learns

one

layer

of

features

at

a

time

We

introduce

this

Bpretraining

-LSB-

procedure

for

binary

data

generalize

it

to

real-valued

data

and

show

that

it

works

well

for

a

variety

of

data

sets

An

ensemble

of

binary

vectors

-LRB-

e.g.

images

-RRB-

can

be

modeled

using

a

two-layer

network

called

a

Brestricted

Boltzmann

machine

-LSB-

-LRB-

RBM

-RRB-

-LRB-

-RRB-

in

which

stochastic

binary

pixels

are

connected

to

stochastic

binary

feature

detectors

using

symmetrically

weighted

connections

The

pixels

correspond

to

Bvisible

-LSB-

units

of

the

RBM

because

their

states

are

observed

the

feature

detectors

correspond

to

Bhidden

-LSB-

units

A

joint

configuration

-LRB-

v

h

-RRB-

of

the

visible

and

hidden

units

has

an

energy

-LRB-

-RRB-

given

by

X

X

E

v

h

j

bi

vi

j

bj

hj

REPORTS

adjusting

the

weights

and

biases

to

lower

the

energy

of

that

image

and

to

raise

the

energy

of

similar

Bconfabulated

-LSB-

images

that

the

network

would

prefer

to

the

real

data

Given

a

training

image

the

binary

state

hj

of

each

feature

detector

j

is

set

to

with

probability

s

-LRB-

bj

P

iviwij

-RRB-

where

s

-LRB-

x

-RRB-

is

the

logistic

function

exp

-LRB-

x

-RRB-

bj

is

the

bias

of

j

vi

is

the

state

of

pixel

i

and

wij

is

the

weight

between

i

and

j

Once

binary

states

have

been

chosen

for

the

hidden

units

a

Bconfabulation

-LSB-

is

produced

by

setting

each

vi

to

with

probability

s

-LRB-

bi

P

jhjwij

-RRB-

where

bi

is

the

bias

of

i

The

states

of

the

hidden

units

are

then

updated

once

more

so

that

they

represent

features

of

the

confabulation

The

change

in

a

weight

is

given

by

Dwij

e

bvi

hj

data

j

bvi

hj

recon

same

learning

rule

is

used

for

the

biases

The

learning

works

well

even

though

it

is

not

exactly

following

the

gradient

of

the

log

probability

of

the

training

data

-LRB-

-RRB-

A

single

layer

of

binary

features

is

not

the

best

way

to

model

the

structure

in

a

set

of

images

After

learning

one

layer

of

feature

detectors

we

can

treat

their

activities

when

they

are

being

driven

by

the

data

as

data

for

learning

a

second

layer

of

features

The

first

layer

of

feature

detectors

then

become

the

visible

units

for

learning

the

next

RBM

This

layer-by-layer

learning

can

be

repeated

as

many

where

e

is

a

learning

rate

bvi

hj

data

is

the

fraction

of

times

that

the

pixel

i

and

feature

detector

j

are

on

together

when

the

feature

detectors

are

being

driven

by

data

and

bvi

hj

recon

is

the

corresponding

fraction

for

confabulations

A

simplified

version

of

the

Fig.

-LRB-

A

-RRB-

The

fraction

of

retrieved

documents

in

the

same

class

as

the

query

when

a

query

document

from

the

test

set

is

used

to

retrieve

other

test

set

documents

averaged

over

all

possible

queries

-LRB-

B

-RRB-

The

codes

produced

by

two-dimensional

LSA

-LRB-

C

-RRB-

The

codes

produced

by

a

autoencoder

JULY

VOL

SCIENCE

Downloaded

from

on

May

Fig.

-LRB-

A

-RRB-

The

twodimensional

codes

for

digits

of

each

class

produced

by

taking

the

first

two

principal

components

of

all

training

images

-LRB-

B

-RRB-

The

two-dimensional

codes

found

by

a

autoencoder

For

an

alternative

visualization

see

-LRB-

-RRB-

REPORTS

times

as

desired

It

can

be

shown

that

adding

an

extra

layer

always

improves

a

lower

bound

on

the

log

probability

that

the

model

assigns

to

the

training

data

provided

the

number

of

feature

detectors

per

layer

does

not

decrease

and

their

weights

are

initialized

correctly

-LRB-

-RRB-

This

bound

does

not

apply

when

the

higher

layers

have

fewer

feature

detectors

but

the

layer-by-layer

learning

algorithm

is

nonetheless

a

very

effective

way

to

pretrain

the

weights

of

a

deep

autoencoder

Each

layer

of

features

captures

strong

high-order

correlations

between

the

activities

of

units

in

the

layer

below

For

a

wide

variety

of

data

sets

this

is

an

efficient

way

to

progressively

reveal

low-dimensional

nonlinear

structure

After

pretraining

multiple

layers

of

feature

detectors

the

model

is

Bunfolded

-LSB-

-LRB-

Fig.

-RRB-

to

produce

encoder

and

decoder

networks

that

initially

use

the

same

weights

The

global

finetuning

stage

then

replaces

stochastic

activities

by

deterministic

real-valued

probabilities

and

uses

backpropagation

through

the

whole

autoencoder

to

fine-tune

the

weights

for

optimal

reconstruction

For

continuous

data

the

hidden

units

of

the

first-level

RBM

remain

binary

but

the

visible

units

are

replaced

by

linear

units

with

Gaussian

noise

-LRB-

-RRB-

If

this

noise

has

unit

variance

the

stochastic

update

rule

for

the

hidden

units

remains

the

same

and

the

update

rule

for

visible

unit

i

is

to

sample

from

P

Gaussian

with

unit

a

variance

and

mean

bi

j

hjwij

In

all

our

experiments

the

visible

units

of

every

RBM

had

real-valued

activities

which

were

in

the

range

for

logistic

units

While

training

higher

level

RBMs

the

visible

units

were

set

to

the

activation

probabilities

of

the

hidden

units

in

the

previous

RBM

but

the

hidden

units

of

every

RBM

except

the

top

one

had

stochastic

binary

values

The

hidden

units

of

the

top

RBM

had

stochastic

real-valued

states

drawn

from

a

unit

variance

Gaussian

whose

mean

was

determined

by

the

input

from

that

logistic

visible

units

This

allowed

the

low-dimensional

codes

to

make

good

use

of

continuous

variables

and

facilitated

comparisons

with

PCA

Details

of

the

pretraining

and

fine-tuning

can

be

found

in

-LRB-

-RRB-

To

demonstrate

that

our

pretraining

algorithm

allows

us

to

fine-tune

deep

networks

efficiently

we

trained

a

very

deep

autoencoder

on

a

synthetic

data

set

containing

images

of

Bcurves

-LSB-

that

were

generated

from

three

randomly

chosen

points

in

two

dimensions

-LRB-

-RRB-

For

this

data

set

the

true

intrinsic

dimensionality

is

known

and

the

relationship

between

the

pixel

intensities

and

the

six

numbers

used

to

generate

them

is

highly

nonlinear

The

pixel

intensities

lie

between

and

and

are

very

non-Gaussian

so

we

used

logistic

output

units

in

the

autoencoder

and

the

fine-tuning

stage

of

the

learning

minimized

the

cross-entropy

error

P

P

E

i

pi

log

pi

i

-LRB-

pi

-RRB-

log

-LRB-

gi

-RRB-

where

p

pi

is

the

intensity

of

pixel

i

and

g

i

is

the

p

intensity

of

its

reconstruction

The

autoencoder

consisted

of

an

encoder

with

layers

of

size

-LRB-

-RRB-

and

a

symmetric

decoder

The

six

units

in

the

code

layer

were

linear

and

all

the

other

units

were

logistic

The

network

was

trained

on

images

and

tested

on

new

images

The

autoencoder

discovered

how

to

convert

each

image

into

six

real

numbers

that

allow

almost

perfect

reconstruction

-LRB-

Fig.

-RRB-

PCA

gave

much

worse

reconstructions

Without

pretraining

the

very

deep

autoencoder

always

reconstructs

the

average

of

the

training

data

even

after

prolonged

fine-tuning

-LRB-

-RRB-

Shallower

autoencoders

with

a

single

hidden

layer

between

the

data

and

the

code

can

learn

without

pretraining

but

pretraining

greatly

reduces

their

total

training

time

-LRB-

-RRB-

When

the

number

of

parameters

is

the

same

deep

autoencoders

can

produce

lower

reconstruction

errors

on

test

data

than

shallow

ones

but

this

advantage

disappears

as

the

number

of

parameters

increases

-LRB-

-RRB-

Next

we

used

a

autoencoder

to

extract

codes

for

all

the

handwritten

digits

in

the

MNIST

training

set

-LRB-

-RRB-

The

Matlab

code

that

we

used

for

the

pretraining

and

fine-tuning

is

available

in

-LRB-

-RRB-

Again

all

units

were

logistic

except

for

the

linear

units

in

the

code

layer

After

fine-tuning

on

all

training

images

the

autoencoder

was

tested

on

new

images

and

produced

much

better

reconstructions

than

did

PCA

-LRB-

Fig.

-RRB-

A

two-dimensional

autoencoder

produced

a

better

visualization

of

the

data

than

did

the

first

two

principal

components

-LRB-

Fig.

-RRB-

We

also

used

a

autoencoder

with

linear

input

units

to

discover

codes

for

grayscale

image

patches

that

were

derived

from

the

Olivetti

face

data

set

-LRB-

-RRB-

The

autoencoder

clearly

outperformed

PCA

-LRB-

Fig.

-RRB-

When

trained

on

documents

autoencoders

produce

codes

that

allow

fast

retrieval

We

represented

each

of

newswire

stories

-LRB-

-RRB-

as

a

vector

of

document-specific

probabilities

of

the

commonest

word

stems

and

we

trained

a

autoencoder

on

half

of

the

stories

with

the

use

P

the

multiclass

of

cross-entropy

error

function

E

i

pi

log

gi

for

p

the

fine-tuning

The

code

units

were

linear

and

the

remaining

hidden

units

were

logistic

When

the

cosine

of

the

angle

between

two

codes

was

used

to

measure

similarity

the

autoencoder

clearly

outperformed

latent

semantic

analysis

-LRB-

LSA

-RRB-

-LRB-

-RRB-

a

well-known

document

retrieval

method

based

on

PCA

-LRB-

Fig.

-RRB-

Autoencoders

-LRB-

-RRB-

also

outperform

local

linear

embedding

a

recent

nonlinear

dimensionality

reduction

algorithm

-LRB-

-RRB-

Layer-by-layer

pretraining

can

also

be

used

for

classification

and

regression

On

a

widely

used

version

of

the

MNIST

handwritten

digit

recogniSCIENCE

VOL

tion

task

the

best

reported

error

rates

are

for

randomly

initialized

backpropagation

and

for

support

vector

machines

After

layer-by-layer

pretraining

in

a

network

backpropagation

using

steepest

descent

and

a

small

learning

rate

achieves

-LRB-

-RRB-

Pretraining

helps

generalization

because

it

ensures

that

most

of

the

information

in

the

weights

comes

from

modeling

the

images

The

very

limited

information

in

the

labels

is

used

only

to

slightly

adjust

the

weights

found

by

pretraining

It

has

been

obvious

since

the

that

backpropagation

through

deep

autoencoders

would

be

very

effective

for

nonlinear

dimensionality

reduction

provided

that

computers

were

fast

enough

data

sets

were

big

enough

and

the

initial

weights

were

close

enough

to

a

good

solution

All

three

conditions

are

now

satisfied

Unlike

nonparametric

methods

-LRB-

-RRB-

autoencoders

give

mappings

in

both

directions

between

the

data

and

code

spaces

and

they

can

be

applied

to

very

large

data

sets

because

both

the

pretraining

and

the

fine-tuning

scale

linearly

in

time

and

space

with

the

number

of

training

cases

References

and

Notes

D.

C.

Plaut

G.

E.

Hinton

Comput

Speech

Lang

-LRB-

-RRB-

D.

DeMers

G.

Cottrell

Advances

in

Neural

Information

Processing

Systems

-LRB-

Morgan

Kaufmann

San

Mateo

CA

-RRB-

pp.

R.

Hecht-Nielsen

Science

-LRB-

-RRB-

N.

Kambhatla

T.

Leen

Neural

Comput

-LRB-

-RRB-

P.

Smolensky

Parallel

Distributed

Processing

Volume

Foundations

D.

E.

Rumelhart

J.

L.

McClelland

Eds

-LRB-

MIT

Press

Cambridge

-RRB-

pp.

G.

E.

Hinton

Neural

Comput

-LRB-

-RRB-

J.

J.

Hopfield

Proc

Natl.

Acad

Sci

U.S.A.

-LRB-

-RRB-

See

supporting

material

on

Science

Online

G.

E.

Hinton

S.

Osindero

Y.

W.

Teh

Neural

Comput

-LRB-

-RRB-

M.

Welling

M.

Rosen-Zvi

G.

Hinton

Advances

in

Neural

Information

Processing

Systems

-LRB-

MIT

Press

Cambridge

MA

-RRB-

pp.

The

MNIST

data

set

is

available

at

The

Olivetti

face

data

set

is

available

at

www

cs.toronto.edu

roweis/data.html

The

Reuter

Corpus

Volume

is

available

at

http

S.

C.

Deerwester

S.

T.

Dumais

T.

K.

Landauer

G.

W.

Furnas

R.

A.

Harshman

J.

Am

Soc

Inf

Sci

-LRB-

-RRB-

S.

T.

Roweis

L.

K.

Saul

Science

-LRB-

-RRB-

J.

A.

Tenenbaum

V.

J.

de

Silva

J.

C.

Langford

Science

-LRB-

-RRB-

We

thank

D.

Rumelhart

M.

Welling

S.

Osindero

and

S.

Roweis

for

helpful

discussions

and

the

Natural

Sciences

and

Engineering

Research

Council

of

Canada

for

funding

G.E.H.

is

a

fellow

of

the

Canadian

Institute

for

Advanced

Research

Supporting

Online

Material

Materials

and

Methods

Figs

to

Matlab

Code

March

accepted

June

science

JULY

Downloaded

from

on

May

O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B B O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

autoencoder
neural networks
network
dimensionality reduction
pca
neural
r

neuralnetwork
machine learning
statistics